[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:00 dist-test-slave-r238 krb5kdc[29673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:00 dist-test-slave-r238 krb5kdc[29673](info): set up 2 sockets
Sep 28 02:46:00 dist-test-slave-r238 krb5kdc[29673](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:46:01.649510 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:01 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:01.777254 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:35653
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:01.886946 29686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:01.887279 29686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:01.887356 29686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:01.891585 29686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:01.891693 29686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:01.891734 29686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:01.891793 29686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:01.891829 29686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:01.897430 29686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:35653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:01.898593 29686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:01.899659 29686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:01.908452 29692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:01.908450 29691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:01.908478 29694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:01.908844 29686 server_base.cc:1034] running on GCE node
I20260928 02:46:01.909467 29686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:01.910619 29686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:01.911806 29686 hybrid_clock.cc:648] HybridClock initialized: now 1790563561911795 us; error 32 us; skew 500 ppm
Sep 28 02:46:01 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:01.914714 29686 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:01.915599 29686 webserver.cc:460] Webserver started at http://127.28.248.190:40613/ using document root <none> and password file <none>
I20260928 02:46:01.916168 29686 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:01.916255 29686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:01.916498 29686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:01.918614 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae"
format_stamp: "Formatted at 2026-09-28 02:46:01 on dist-test-slave-r238"
server_key: "dc6a8cac87188ac3a1c7cfd7bdd3d97c"
server_key_iv: "ac3c24f56392181f9705478f75c2cdd3"
server_key_version: "encryptionkey@0"
I20260928 02:46:01.919204 29686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae"
format_stamp: "Formatted at 2026-09-28 02:46:01 on dist-test-slave-r238"
server_key: "dc6a8cac87188ac3a1c7cfd7bdd3d97c"
server_key_iv: "ac3c24f56392181f9705478f75c2cdd3"
server_key_version: "encryptionkey@0"
I20260928 02:46:01.922986 29686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:01.925459 29701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:01.926813 29686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 02:46:01.926956 29686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae"
format_stamp: "Formatted at 2026-09-28 02:46:01 on dist-test-slave-r238"
server_key: "dc6a8cac87188ac3a1c7cfd7bdd3d97c"
server_key_iv: "ac3c24f56392181f9705478f75c2cdd3"
server_key_version: "encryptionkey@0"
I20260928 02:46:01.927075 29686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:01.937785 29686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:01.941402 29686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:01.941613 29686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:01.948779 29686 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:35653
I20260928 02:46:01.948832 29753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:35653 every 8 connection(s)
I20260928 02:46:01.949760 29686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:01.952739 29754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:01.953963 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 29686
I20260928 02:46:01.954128 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:01.955762 29666 external_mini_cluster.cc:1428] Setting key f640a686ad32a0e98bede5fd97f9f356
I20260928 02:46:01.959759 29754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae: Bootstrap starting.
I20260928 02:46:01.962549 29754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:01.963542 29754 log.cc:824] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:01.965715 29754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae: No bootstrap required, opened a new log
I20260928 02:46:01.970155 29754 raft_consensus.cc:348] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35653 } }
I20260928 02:46:01.970456 29754 raft_consensus.cc:374] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:01.970513 29754 raft_consensus.cc:729] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55f7f5c7de94d8793e8d58ba9fe7fae, State: Initialized, Role: FOLLOWER
I20260928 02:46:01.970988 29754 consensus_queue.cc:260] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35653 } }
I20260928 02:46:01.971136 29754 raft_consensus.cc:388] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:01.971197 29754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:01.971297 29754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:01.972270 29754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35653 } }
I20260928 02:46:01.972656 29754 leader_election.cc:302] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d55f7f5c7de94d8793e8d58ba9fe7fae; no voters:
I20260928 02:46:01.973027 29754 leader_election.cc:288] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:01.973165 29759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:01.973398 29759 raft_consensus.cc:686] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [term 1 LEADER]: Becoming Leader. State: Replica: d55f7f5c7de94d8793e8d58ba9fe7fae, State: Running, Role: LEADER
I20260928 02:46:01.973699 29759 consensus_queue.cc:237] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35653 } }
I20260928 02:46:01.974418 29754 sys_catalog.cc:564] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:01.975437 29761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d55f7f5c7de94d8793e8d58ba9fe7fae. Latest consensus state: current_term: 1 leader_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35653 } } }
I20260928 02:46:01.975437 29760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d55f7f5c7de94d8793e8d58ba9fe7fae" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35653 } } }
I20260928 02:46:01.975584 29760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:01.975584 29761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:01.976011 29768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 02:46:01 dist-test-slave-r238 krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:01.979259 29768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:01.983717 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:01.959538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.28.248.190:35653)
0928 02:46:01.960022 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:01.960040 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:01.961149 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 02:46:01.965517 (+ 4368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:01.965534 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:01.965904 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:01.966173 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:01.969047 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:01.970034 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:01.971725 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:01.972171 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:01.972315 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:01.977527 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:01.977533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:01.977549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:01.977579 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:01.979784 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:01.980439 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:01.980445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:01.980450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:01.980593 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:01.981101 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:01.981105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:01.981106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:01.981616 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:01.981825 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:01.982486 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 02:46:01.982771 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260928 02:46:01.986425 29768 catalog_manager.cc:1347] Generated new cluster ID: bd2d8052648941fb9bf55ec3c6ce5c15
I20260928 02:46:01.986555 29768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:01.999734 29768 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:02.000789 29768 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:02.017952 29768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d55f7f5c7de94d8793e8d58ba9fe7fae: Generated new TSK 0
I20260928 02:46:02.018909 29768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:02.051278 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35653
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:02.160557 29780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:02.160880 29780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:02.160964 29780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:02.164958 29780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:02.165036 29780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:02.165127 29780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:02.170230 29780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:02.171272 29780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:02.172274 29780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:02.181924 29788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:02.181945 29785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:02.181993 29780 server_base.cc:1034] running on GCE node
W20260928 02:46:02.181945 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:02.182642 29780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:02.183247 29780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:02.184432 29780 hybrid_clock.cc:648] HybridClock initialized: now 1790563562184417 us; error 26 us; skew 500 ppm
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:02.187212 29780 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:02.188083 29780 webserver.cc:460] Webserver started at http://127.28.248.129:40121/ using document root <none> and password file <none>
I20260928 02:46:02.188690 29780 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:02.188778 29780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:02.189008 29780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:02.190984 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "b40cf71e840e4833b13c7eebbbec25e0"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "f7ab026c7664a50720842b61b834e62e"
server_key_iv: "a326145c9dc3695a20ae4c64e091ea88"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.191524 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "b40cf71e840e4833b13c7eebbbec25e0"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "f7ab026c7664a50720842b61b834e62e"
server_key_iv: "a326145c9dc3695a20ae4c64e091ea88"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.195055 29780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:02.197350 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.198483 29780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:02.198601 29780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "b40cf71e840e4833b13c7eebbbec25e0"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "f7ab026c7664a50720842b61b834e62e"
server_key_iv: "a326145c9dc3695a20ae4c64e091ea88"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.198701 29780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:02.219746 29780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:02.223045 29780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:02.223265 29780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:02.223886 29780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:02.224725 29780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:02.224771 29780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.224809 29780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:02.224871 29780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.233492 29780 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:42725
I20260928 02:46:02.233513 29908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:42725 every 8 connection(s)
I20260928 02:46:02.234335 29780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:02.237532 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 29780
I20260928 02:46:02.237650 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:02.237923 29666 external_mini_cluster.cc:1428] Setting key dd8128465c4e8f2d0aae014b921ecc04
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:02.248028 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.236068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:33563 (local address 127.28.248.190:35653)
0928 02:46:02.236191 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:02.236195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:02.237244 (+ 1049us) server_negotiation.cc:407] Connection header received
0928 02:46:02.238259 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:02.238262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:02.238321 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:02.238430 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:02.239533 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.240066 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.241256 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.241497 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.241608 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.244365 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:02.244368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:02.244370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:02.244392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:02.245901 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.246488 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.246490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.246492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.246561 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.246955 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.246957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.246958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.247116 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:02.247188 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:02.247787 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 02:46:02.247906 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:46:02.248919 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.236439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:35653 (local address 127.28.248.129:33563)
0928 02:46:02.237108 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:02.237147 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:02.238065 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:02.238604 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:02.238611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:02.238944 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:02.239396 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:02.239408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.240189 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.240192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:02.241135 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:02.241144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.241642 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.241646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:02.241803 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.242400 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:02.242416 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:02.244198 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:02.246020 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:02.246028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:02.246043 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:02.246390 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:02.246654 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:02.246659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:02.246664 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:02.246861 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:02.247220 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:02.247230 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:02.247552 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 02:46:02.247744 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:46:02.247998 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:02.250108 29909 heartbeater.cc:347] Connected to a master server at 127.28.248.190:35653
I20260928 02:46:02.250349 29909 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:02.250972 29909 heartbeater.cc:511] Master 127.28.248.190:35653 requested a full tablet report, sending...
I20260928 02:46:02.252823 29718 ts_manager.cc:194] Registered new tserver with Master: b40cf71e840e4833b13c7eebbbec25e0 (127.28.248.129:42725)
I20260928 02:46:02.253940 29718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:33563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:02.267705 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35653
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:02.378433 29914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:02.378664 29914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:02.378713 29914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:02.382714 29914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:02.382802 29914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:02.382907 29914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:02.388160 29914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:02.389249 29914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:02.390307 29914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:02.399554 29922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:02.399564 29920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:02.399564 29919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:02.399974 29914 server_base.cc:1034] running on GCE node
I20260928 02:46:02.400363 29914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:02.401046 29914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:02.402185 29914 hybrid_clock.cc:648] HybridClock initialized: now 1790563562402175 us; error 49 us; skew 500 ppm
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:02.404924 29914 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:02.405735 29914 webserver.cc:460] Webserver started at http://127.28.248.130:41191/ using document root <none> and password file <none>
I20260928 02:46:02.406273 29914 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:02.406333 29914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:02.406575 29914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:02.408531 29914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "f353430c4ef6458c91d31d5258e30920"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "e92999702da9f5b3c1b376fd9f753e8d"
server_key_iv: "b84acb98e8dd0acf2e35510537496f4f"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.409140 29914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "f353430c4ef6458c91d31d5258e30920"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "e92999702da9f5b3c1b376fd9f753e8d"
server_key_iv: "b84acb98e8dd0acf2e35510537496f4f"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.412722 29914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:02.415006 29929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.416186 29914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:02.416327 29914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "f353430c4ef6458c91d31d5258e30920"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "e92999702da9f5b3c1b376fd9f753e8d"
server_key_iv: "b84acb98e8dd0acf2e35510537496f4f"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.416460 29914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:02.427670 29914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:02.430644 29914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:02.430833 29914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:02.431464 29914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:02.432238 29914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:02.432309 29914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.432377 29914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:02.432415 29914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.441102 29914 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:34075
I20260928 02:46:02.441124 30042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:34075 every 8 connection(s)
I20260928 02:46:02.441953 29914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:02.443816 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 29914
I20260928 02:46:02.443986 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:02.444252 29666 external_mini_cluster.cc:1428] Setting key c303b35a0783df99eb995cd7b55f14a7
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:46:02.456990 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.443878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:34009 (local address 127.28.248.190:35653)
0928 02:46:02.444071 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:02.444075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:02.445254 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 02:46:02.446190 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:02.446196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:02.446256 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:02.446357 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:02.447567 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.448434 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.449789 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.450183 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.450318 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.453198 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:02.453205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:02.453212 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:02.453243 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:02.454742 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.455343 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.455346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.455347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.455423 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.455768 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.455770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.455771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.455951 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:02.456026 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:02.456589 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 02:46:02.456749 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 02:46:02.457975 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.444317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:35653 (local address 127.28.248.130:34009)
0928 02:46:02.445079 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:02.445133 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:02.445976 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:02.446517 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:02.446524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:02.446898 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:02.447416 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:02.447430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.448601 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.448608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:02.449661 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:02.449672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.450364 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.450368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:02.450543 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.451252 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:02.451270 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:02.453009 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:02.454894 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:02.454904 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:02.454926 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:02.455240 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:02.455510 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:02.455512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:02.455514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:02.455661 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:02.456071 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:02.456085 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:02.456359 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 02:46:02.456524 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:46:02.457034 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":148,"threads_started":1}
I20260928 02:46:02.459225 30043 heartbeater.cc:347] Connected to a master server at 127.28.248.190:35653
I20260928 02:46:02.459489 30043 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:02.459990 30043 heartbeater.cc:511] Master 127.28.248.190:35653 requested a full tablet report, sending...
I20260928 02:46:02.461170 29718 ts_manager.cc:194] Registered new tserver with Master: f353430c4ef6458c91d31d5258e30920 (127.28.248.130:34075)
I20260928 02:46:02.462056 29718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:34009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:02.472222 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35653
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:02.583818 30048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:02.584092 30048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:02.584170 30048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:02.588295 30048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:02.588372 30048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:02.588457 30048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:02.593673 30048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35653
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:02.594744 30048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:02.595752 30048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:02.605544 30054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:02.605571 30053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:02.605556 30056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:02.605894 30048 server_base.cc:1034] running on GCE node
I20260928 02:46:02.606297 30048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:02.606962 30048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:02.608147 30048 hybrid_clock.cc:648] HybridClock initialized: now 1790563562608129 us; error 33 us; skew 500 ppm
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:02.611105 30048 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:02.612017 30048 webserver.cc:460] Webserver started at http://127.28.248.131:41949/ using document root <none> and password file <none>
I20260928 02:46:02.612670 30048 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:02.612761 30048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:02.613003 30048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:02.615074 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "18bb141c633b4f1fba3e40d8069b4197"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "28bcb6616d57b0733dd1d5a58897044f"
server_key_iv: "6dfb7dcfe6fc9e26771c9e3573f79617"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.615700 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "18bb141c633b4f1fba3e40d8069b4197"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "28bcb6616d57b0733dd1d5a58897044f"
server_key_iv: "6dfb7dcfe6fc9e26771c9e3573f79617"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.619521 30048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:02.622272 30063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.623601 30048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:02.623817 30048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "18bb141c633b4f1fba3e40d8069b4197"
format_stamp: "Formatted at 2026-09-28 02:46:02 on dist-test-slave-r238"
server_key: "28bcb6616d57b0733dd1d5a58897044f"
server_key_iv: "6dfb7dcfe6fc9e26771c9e3573f79617"
server_key_version: "encryptionkey@0"
I20260928 02:46:02.623946 30048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:02.634191 30048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:02.637108 30048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:02.637331 30048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:02.637909 30048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:02.638736 30048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:02.638810 30048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.638885 30048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:02.638948 30048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:02.648725 30048 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:44519
I20260928 02:46:02.648746 30176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:44519 every 8 connection(s)
I20260928 02:46:02.649664 30048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:02.658565 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30048
I20260928 02:46:02.658695 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:02.658988 29666 external_mini_cluster.cc:1428] Setting key 02969c4b477d9a5917fbff8fa2bd2e65
I20260928 02:46:02.662957 29756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.651518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54477 (local address 127.28.248.190:35653)
0928 02:46:02.651650 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:02.651654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:02.652750 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 02:46:02.653676 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:02.653679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:02.653734 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:02.653823 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:02.654969 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.655508 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.656655 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.656883 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.656997 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.659500 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:02.659504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:02.659507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:02.659539 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:02.661181 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.661705 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.661709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.661710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.661788 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.662062 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.662064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.662065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.662246 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:02.662322 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:02.662744 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 02:46:02.662836 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:46:02.663681 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.651928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:35653 (local address 127.28.248.131:54477)
0928 02:46:02.652563 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:02.652602 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:02.653481 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:02.654012 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:02.654019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:02.654364 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:02.654818 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:02.654830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.655635 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.655645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:02.656506 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:02.656514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.657031 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.657035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:02.657202 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.657766 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:02.657781 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:02.659333 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:02.661298 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:02.661303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:02.661319 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:02.661630 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:02.661879 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:02.661881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:02.661882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:02.662004 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:02.662327 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:02.662332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:02.662539 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 02:46:02.662686 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:46:02.662923 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":114,"threads_started":1}
I20260928 02:46:02.664709 30177 heartbeater.cc:347] Connected to a master server at 127.28.248.190:35653
I20260928 02:46:02.664915 30177 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:02.665349 30177 heartbeater.cc:511] Master 127.28.248.190:35653 requested a full tablet report, sending...
I20260928 02:46:02.666281 29718 ts_manager.cc:194] Registered new tserver with Master: 18bb141c633b4f1fba3e40d8069b4197 (127.28.248.131:44519)
I20260928 02:46:02.666795 29718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:54477
I20260928 02:46:02.673496 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:02.700700 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.690579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.28.248.129:42725)
0928 02:46:02.690926 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:02.690935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:02.690960 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:46:02.691011 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:02.691015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:02.691219 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:02.691352 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:02.692070 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.692982 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.693865 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.694345 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.694432 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.697060 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:02.697068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:02.697080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:02.697103 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:02.698636 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.699238 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.699243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.699245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.699338 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.699612 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.699614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.699615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.699879 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:02.700032 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:02.700260 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:46:02.700424 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
W20260928 02:46:02.701773 29823 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51136
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:02.728720 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:02.720230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.28.248.129:42725)
0928 02:46:02.720378 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:02.720383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:02.720485 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:46:02.720669 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:02.720673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:02.720728 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:02.720860 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:02.721532 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.722066 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.722890 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:02.723137 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:02.723246 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:02.725608 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:02.725612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:02.725617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:02.725644 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:02.726893 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.727295 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.727299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.727301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.727384 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:02.727653 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:02.727657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:02.727659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:02.727859 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:02.727988 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:02.728254 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:46:02.728425 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:46:02.729890 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 29780
I20260928 02:46:02.737303 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 29914
I20260928 02:46:02.743160 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30048
I20260928 02:46:02.749204 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 29686
2026-09-28T02:46:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2202 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[30191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[30191](info): set up 2 sockets
Sep 28 02:46:02 dist-test-slave-r238 krb5kdc[30191](info): commencing operation
krb5kdc: starting...
W20260928 02:46:03.888454 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:03 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:04.011336 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46205
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:46205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:04.119370 30204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:04.119683 30204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:04.119764 30204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:04.123750 30204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:04.123849 30204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:04.123886 30204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:04.123927 30204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:04.123961 30204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:04.129196 30204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:46205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:04.130247 30204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:04.131279 30204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:04.139652 30209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.139624 30212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.139608 30210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:04.140106 30204 server_base.cc:1034] running on GCE node
I20260928 02:46:04.140790 30204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:04.141945 30204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:04.143123 30204 hybrid_clock.cc:648] HybridClock initialized: now 1790563564143097 us; error 36 us; skew 500 ppm
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:04.146144 30204 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:04.147094 30204 webserver.cc:460] Webserver started at http://127.28.248.190:45515/ using document root <none> and password file <none>
I20260928 02:46:04.147679 30204 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:04.147778 30204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:04.148023 30204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:04.150218 30204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "b3897be8f2bd4fd2a428b68a7c27a07e"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "bf4c4575cf9c595ae2c6cbf9a465afa4"
server_key_iv: "8bbe3f02d61b07b81c5da5354508d6c9"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.150862 30204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "b3897be8f2bd4fd2a428b68a7c27a07e"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "bf4c4575cf9c595ae2c6cbf9a465afa4"
server_key_iv: "8bbe3f02d61b07b81c5da5354508d6c9"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.154986 30204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:46:04.157670 30219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.159010 30204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:04.159170 30204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "b3897be8f2bd4fd2a428b68a7c27a07e"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "bf4c4575cf9c595ae2c6cbf9a465afa4"
server_key_iv: "8bbe3f02d61b07b81c5da5354508d6c9"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.159291 30204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:04.184676 30204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:04.188171 30204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:04.188388 30204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:04.195276 30204 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:46205
I20260928 02:46:04.195328 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:46205 every 8 connection(s)
I20260928 02:46:04.196256 30204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:04.197162 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30204
I20260928 02:46:04.197286 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:04.197535 29666 external_mini_cluster.cc:1428] Setting key 95666f5fe5b67370c8ece1d38e4f858e
I20260928 02:46:04.200070 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:04.206532 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e: Bootstrap starting.
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:04.208882 30272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:04.209561 30272 log.cc:824] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:04.211380 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e: No bootstrap required, opened a new log
I20260928 02:46:04.212322 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.198812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.28.248.190:46205)
0928 02:46:04.199213 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.199229 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.199267 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:46:04.200020 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.200039 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.200442 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.200802 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:04.201597 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.202741 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.203926 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.204468 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.204602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.207473 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:04.207485 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:04.207500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:04.207526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:04.209330 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.209782 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.209787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.209791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.209896 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.210203 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.210208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.210210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.210585 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:04.210796 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.211084 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.211353 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260928 02:46:04.214694 30272 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46205 } }
I20260928 02:46:04.214952 30272 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:04.215040 30272 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3897be8f2bd4fd2a428b68a7c27a07e, State: Initialized, Role: FOLLOWER
I20260928 02:46:04.215426 30272 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46205 } }
I20260928 02:46:04.215567 30272 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:04.215652 30272 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:04.215751 30272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:04.216558 30272 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46205 } }
I20260928 02:46:04.216949 30272 leader_election.cc:302] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3897be8f2bd4fd2a428b68a7c27a07e; no voters:
I20260928 02:46:04.217301 30272 leader_election.cc:288] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:04.217383 30277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:04.217548 30277 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [term 1 LEADER]: Becoming Leader. State: Replica: b3897be8f2bd4fd2a428b68a7c27a07e, State: Running, Role: LEADER
I20260928 02:46:04.217823 30277 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46205 } }
I20260928 02:46:04.218519 30272 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:04.218952 30277 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46205 } } }
I20260928 02:46:04.219059 30277 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:04.219246 30277 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3897be8f2bd4fd2a428b68a7c27a07e. Latest consensus state: current_term: 1 leader_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3897be8f2bd4fd2a428b68a7c27a07e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46205 } } }
I20260928 02:46:04.219347 30277 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:04.219532 30281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:04.222282 30281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:04.227509 30281 catalog_manager.cc:1347] Generated new cluster ID: eba169c9217d4c818451f259487aa214
I20260928 02:46:04.227571 30281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:04.237766 30281 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:04.238932 30281 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:04.247514 30281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3897be8f2bd4fd2a428b68a7c27a07e: Generated new TSK 0
I20260928 02:46:04.248325 30281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:04.280705 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46205
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:04.391997 30298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:04.392278 30298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:04.392355 30298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:04.396306 30298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:04.396394 30298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:04.396486 30298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:04.401614 30298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:04.402606 30298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:04.403601 30298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:46:04.413129 30298 server_base.cc:1034] running on GCE node
W20260928 02:46:04.413111 30304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.413082 30303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.413082 30306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:04.413741 30298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:04.414315 30298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:04.415490 30298 hybrid_clock.cc:648] HybridClock initialized: now 1790563564415460 us; error 40 us; skew 500 ppm
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:04.418313 30298 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:04.419246 30298 webserver.cc:460] Webserver started at http://127.28.248.129:46717/ using document root <none> and password file <none>
I20260928 02:46:04.419952 30298 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:04.420054 30298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:04.420264 30298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:04.422349 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "05e9b43ceabb479c9c2ab9a131babca8"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "aa48ee5045f3420f512cbd280803a489"
server_key_iv: "3ef5718a85e3384fa0c45bdb0b7e6968"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.422901 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "05e9b43ceabb479c9c2ab9a131babca8"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "aa48ee5045f3420f512cbd280803a489"
server_key_iv: "3ef5718a85e3384fa0c45bdb0b7e6968"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.426574 30298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:04.429097 30313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.430487 30298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:04.430639 30298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "05e9b43ceabb479c9c2ab9a131babca8"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "aa48ee5045f3420f512cbd280803a489"
server_key_iv: "3ef5718a85e3384fa0c45bdb0b7e6968"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.430770 30298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:04.467880 30298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:04.470995 30298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:04.471223 30298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:04.471869 30298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:04.472728 30298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:04.472801 30298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.472862 30298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:04.472901 30298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.481578 30298 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:39143
I20260928 02:46:04.481612 30426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:39143 every 8 connection(s)
I20260928 02:46:04.482460 30298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:04.487267 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30298
I20260928 02:46:04.487387 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:04.487656 29666 external_mini_cluster.cc:1428] Setting key 8062c47a6fd968257b06970222298ea3
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:04.497565 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.484195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:51243 (local address 127.28.248.190:46205)
0928 02:46:04.484346 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.484351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.485251 (+ 900us) server_negotiation.cc:407] Connection header received
0928 02:46:04.486528 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.486531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.486577 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.486655 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:04.488182 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.488937 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.490652 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.491040 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.491141 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.493892 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:04.493896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:04.493899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:04.493925 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:04.495518 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.496028 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.496034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.496036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.496113 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.496446 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.496451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.496452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.496685 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:04.496790 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.497278 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.497403 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:04.498390 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.484662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46205 (local address 127.28.248.129:51243)
0928 02:46:04.485096 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:04.485139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:04.486315 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:04.486813 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:04.486819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:04.487304 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:04.487978 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.488000 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.489063 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.489066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.490500 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.490514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.491209 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.491215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.491450 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.492076 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:04.492101 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:04.493721 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:04.495639 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:04.495645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:04.495659 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:04.495924 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:04.496208 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:04.496211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:04.496213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:04.496351 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:04.496799 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:04.496808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:04.497044 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 02:46:04.497242 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 02:46:04.497550 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260928 02:46:04.499588 30427 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:04.499878 30427 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:04.500360 30427 heartbeater.cc:511] Master 127.28.248.190:46205 requested a full tablet report, sending...
I20260928 02:46:04.501884 30236 ts_manager.cc:194] Registered new tserver with Master: 05e9b43ceabb479c9c2ab9a131babca8 (127.28.248.129:39143)
I20260928 02:46:04.502974 30236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:51243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:04.516016 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46205
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:04.624346 30432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:04.624588 30432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:04.624696 30432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:04.628587 30432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:04.628695 30432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:04.628800 30432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:04.633877 30432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:04.634843 30432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:04.635845 30432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:04.645174 30440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.645203 30437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.645213 30438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:04.645340 30432 server_base.cc:1034] running on GCE node
I20260928 02:46:04.645864 30432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:04.646435 30432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:04.647645 30432 hybrid_clock.cc:648] HybridClock initialized: now 1790563564647635 us; error 29 us; skew 500 ppm
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:04.650480 30432 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:04.651332 30432 webserver.cc:460] Webserver started at http://127.28.248.130:34717/ using document root <none> and password file <none>
I20260928 02:46:04.651891 30432 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:04.651983 30432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:04.652220 30432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:04.654310 30432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "01474a4664c84905aa6cad537ecbe4f8"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "d76b17c5c11f51d1079bb648b7b46441"
server_key_iv: "339d415a05eb208153ba21718fbc563b"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.654920 30432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "01474a4664c84905aa6cad537ecbe4f8"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "d76b17c5c11f51d1079bb648b7b46441"
server_key_iv: "339d415a05eb208153ba21718fbc563b"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.658763 30432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:46:04.661129 30447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.662454 30432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:04.662585 30432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "01474a4664c84905aa6cad537ecbe4f8"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "d76b17c5c11f51d1079bb648b7b46441"
server_key_iv: "339d415a05eb208153ba21718fbc563b"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.662693 30432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:04.683385 30432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:04.686401 30432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:04.686625 30432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:04.687260 30432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:04.688120 30432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:04.688206 30432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.688284 30432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:04.688331 30432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.697956 30432 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:40751
I20260928 02:46:04.697978 30560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:40751 every 8 connection(s)
I20260928 02:46:04.698853 30432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:04.702220 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30432
I20260928 02:46:04.702320 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:04.702623 29666 external_mini_cluster.cc:1428] Setting key fd413defeb357bfb2db19c629d9e4e6b
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:04.712548 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.700701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:49697 (local address 127.28.248.190:46205)
0928 02:46:04.700846 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.700850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.701667 (+ 817us) server_negotiation.cc:407] Connection header received
0928 02:46:04.703093 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.703096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.703151 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.703234 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:04.704376 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.704970 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.706086 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.706309 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.706429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.708891 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:04.708895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:04.708899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:04.708927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:04.710444 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.710965 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.710970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.710972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.711054 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.711485 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.711488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.711490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.711665 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:04.711722 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.712252 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.712378 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:46:04.713428 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.701072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46205 (local address 127.28.248.130:49697)
0928 02:46:04.701518 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:04.701565 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:04.702891 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:04.703404 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:04.703411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:04.703754 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:04.704230 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.704242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.705088 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.705094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.705964 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.705973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.706463 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.706468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.706634 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.707226 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:04.707243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:04.708734 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:04.710557 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:04.710563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:04.710578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:04.710830 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:04.711168 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:04.711173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:04.711175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:04.711325 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:04.711788 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:04.711792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:04.712021 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 02:46:04.712211 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:46:04.712525 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:46:04.714684 30561 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46205
I20260928 02:46:04.714915 30561 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:04.715397 30561 heartbeater.cc:511] Master 127.28.248.190:46205 requested a full tablet report, sending...
I20260928 02:46:04.716430 30236 ts_manager.cc:194] Registered new tserver with Master: 01474a4664c84905aa6cad537ecbe4f8 (127.28.248.130:40751)
I20260928 02:46:04.717047 30236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:49697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:04.729415 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46205
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:04.839650 30566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:04.839897 30566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:04.839939 30566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:04.844046 30566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:04.844099 30566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:04.844172 30566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:04.849220 30566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:04.850116 30566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:04.851029 30566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:04.860275 30572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.860316 30571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:04.860291 30574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:04.860366 30566 server_base.cc:1034] running on GCE node
I20260928 02:46:04.860996 30566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:04.861594 30566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:04.862789 30566 hybrid_clock.cc:648] HybridClock initialized: now 1790563564862767 us; error 34 us; skew 500 ppm
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:04.865685 30566 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:04.866581 30566 webserver.cc:460] Webserver started at http://127.28.248.131:45179/ using document root <none> and password file <none>
I20260928 02:46:04.867139 30566 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:04.867203 30566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:04.867367 30566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:04.869329 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "07b1512597db42c79e075923a4c3c6ef"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "0a45a5dc61115fac9fa0a9a3ec797da1"
server_key_iv: "94d7241b38e4191d9ee7430625149de9"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.869812 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "07b1512597db42c79e075923a4c3c6ef"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "0a45a5dc61115fac9fa0a9a3ec797da1"
server_key_iv: "94d7241b38e4191d9ee7430625149de9"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.873363 30566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:04.875638 30581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.876886 30566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:04.876999 30566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "07b1512597db42c79e075923a4c3c6ef"
format_stamp: "Formatted at 2026-09-28 02:46:04 on dist-test-slave-r238"
server_key: "0a45a5dc61115fac9fa0a9a3ec797da1"
server_key_iv: "94d7241b38e4191d9ee7430625149de9"
server_key_version: "encryptionkey@0"
I20260928 02:46:04.877094 30566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:04.896978 30566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:04.900135 30566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:04.900332 30566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:04.900933 30566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:04.901695 30566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:04.901739 30566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.901808 30566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:04.901847 30566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:04.910626 30566 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:44063
I20260928 02:46:04.910652 30694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:44063 every 8 connection(s)
I20260928 02:46:04.911497 30566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:04.915684 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30566
I20260928 02:46:04.915853 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:04.916220 29666 external_mini_cluster.cc:1428] Setting key 206f8ff64b3b7586b58a8389c653578b
Sep 28 02:46:04 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:04.926045 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.913426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54883 (local address 127.28.248.190:46205)
0928 02:46:04.913567 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.913571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.914441 (+ 870us) server_negotiation.cc:407] Connection header received
0928 02:46:04.915723 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.915726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.915781 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.915863 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:04.917667 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.918547 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.919835 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.920046 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.920125 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.922644 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:04.922647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:04.922649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:04.922672 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:04.924102 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.924598 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.924601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.924602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.924744 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.925108 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.925111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.925112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.925294 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:04.925372 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.925828 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.925931 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:04.926914 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.913857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46205 (local address 127.28.248.131:54883)
0928 02:46:04.914296 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:04.914335 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:04.915513 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:04.916085 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:04.916096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:04.916717 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:04.917463 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.917490 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.918737 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.918740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.919732 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.919741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.920213 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.920217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.920380 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.920990 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:04.921007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:04.922489 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:04.924221 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:04.924225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:04.924236 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:04.924509 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:04.924851 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:04.924855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:04.924860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:04.925024 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:04.925388 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:04.925393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:04.925619 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 02:46:04.925787 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:46:04.926050 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":130,"threads_started":1}
I20260928 02:46:04.928120 30695 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46205
I20260928 02:46:04.928352 30695 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:04.928892 30695 heartbeater.cc:511] Master 127.28.248.190:46205 requested a full tablet report, sending...
I20260928 02:46:04.930016 30236 ts_manager.cc:194] Registered new tserver with Master: 07b1512597db42c79e075923a4c3c6ef (127.28.248.131:44063)
I20260928 02:46:04.930584 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:04.930820 30236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:54883
I20260928 02:46:04.941107 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.934103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.28.248.190:46205)
0928 02:46:04.934245 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.934250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.934453 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:46:04.934541 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.934543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.934577 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.934639 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:04.935348 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.935819 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.936679 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.936970 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.937062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.938285 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:04.938287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:04.938289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:04.938309 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:04.939674 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.940045 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.940048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.940049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.940115 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:04.940332 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:04.940334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:04.940335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:04.940517 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:04.940685 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.940915 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.941001 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:46:04.945618 30236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:04.948450 30236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:04.963105 30706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.956924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39143 (local address 127.0.0.1:42836)
0928 02:46:04.957274 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:04.957329 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:04.957459 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:04.958333 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:04.958339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:04.958365 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:04.958509 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.958518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.959905 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.959909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.961462 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.961473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.962532 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.962536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.962641 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.962697 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:46:04.962805 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:46:04.962927 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260928 02:46:04.963296 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.957456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:44063 (local address 127.0.0.1:45246)
0928 02:46:04.958195 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:04.958208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:04.958461 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:04.959182 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:04.959185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:04.959201 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:04.959359 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.959366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.960657 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.960660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.961705 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.961715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.962888 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.962891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.962979 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.963014 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:46:04.963089 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:46:04.963154 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":30,"thread_start_us":112,"threads_started":1}
I20260928 02:46:04.963402 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.957902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.28.248.131:44063)
0928 02:46:04.958710 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.958725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.958746 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:46:04.958817 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.958824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.959033 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.959486 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:04.959627 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.960477 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.961829 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.962700 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.962878 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.962993 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.963132 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.963246 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":325,"threads_started":1}
I20260928 02:46:04.963713 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.957113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.28.248.129:39143)
0928 02:46:04.957454 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.957471 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.957496 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:46:04.957694 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.957702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.957920 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.958097 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:04.958628 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.959757 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.961591 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.962405 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.962532 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.962665 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.963427 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.963535 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":61,"threads_started":1}
I20260928 02:46:04.964691 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.957813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40751 (local address 127.0.0.1:44900)
0928 02:46:04.958569 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:04.958586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:04.958692 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:04.959116 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:04.959119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:04.959137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:04.959345 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.959362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.960871 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.960874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.961949 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:04.961958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.963267 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.963271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:04.963370 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.963394 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:46:04.963709 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 02:46:04.964549 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":51,"thread_start_us":85,"threads_started":1}
I20260928 02:46:04.965131 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:04.957822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.28.248.130:40751)
0928 02:46:04.958515 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:04.958529 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:04.958673 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:46:04.958817 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:04.958821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:04.958967 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:04.959129 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:04.959478 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.960565 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.962071 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:04.962967 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:04.963095 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:04.963210 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:04.964939 (+ 1729us) server_negotiation.cc:299] Negotiation successful
0928 02:46:04.965025 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":115,"threads_started":1}
I20260928 02:46:04.966691 30629 tablet_service.cc:1467] Processing CreateTablet for tablet ccac886059cf4f9f9fafba5f5ed016a8 (DEFAULT_TABLE table=test-table [id=5e552726ea784127a95f1eb4a6de713c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:04.966688 30361 tablet_service.cc:1467] Processing CreateTablet for tablet ccac886059cf4f9f9fafba5f5ed016a8 (DEFAULT_TABLE table=test-table [id=5e552726ea784127a95f1eb4a6de713c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:04.967840 30629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac886059cf4f9f9fafba5f5ed016a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:04.967835 30361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac886059cf4f9f9fafba5f5ed016a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:04.967621 30495 tablet_service.cc:1467] Processing CreateTablet for tablet ccac886059cf4f9f9fafba5f5ed016a8 (DEFAULT_TABLE table=test-table [id=5e552726ea784127a95f1eb4a6de713c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:04.968650 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccac886059cf4f9f9fafba5f5ed016a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:04.974457 30712 tablet_bootstrap.cc:492] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8: Bootstrap starting.
I20260928 02:46:04.975868 30714 tablet_bootstrap.cc:492] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8: Bootstrap starting.
I20260928 02:46:04.976935 30713 tablet_bootstrap.cc:492] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef: Bootstrap starting.
I20260928 02:46:04.977068 30712 tablet_bootstrap.cc:654] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:04.977855 30714 tablet_bootstrap.cc:654] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:04.977833 30712 log.cc:824] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:04.978987 30713 tablet_bootstrap.cc:654] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:04.979007 30714 log.cc:824] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:04.979882 30712 tablet_bootstrap.cc:492] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8: No bootstrap required, opened a new log
I20260928 02:46:04.980083 30712 ts_tablet_manager.cc:1397] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:46:04.980147 30713 log.cc:824] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:04.981003 30714 tablet_bootstrap.cc:492] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8: No bootstrap required, opened a new log
I20260928 02:46:04.981196 30714 ts_tablet_manager.cc:1397] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260928 02:46:04.982084 30713 tablet_bootstrap.cc:492] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef: No bootstrap required, opened a new log
I20260928 02:46:04.982254 30713 ts_tablet_manager.cc:1397] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:46:04.983541 30712 raft_consensus.cc:348] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:04.983801 30712 raft_consensus.cc:374] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:04.983853 30712 raft_consensus.cc:729] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01474a4664c84905aa6cad537ecbe4f8, State: Initialized, Role: FOLLOWER
I20260928 02:46:04.984297 30712 consensus_queue.cc:260] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:04.984938 30714 raft_consensus.cc:348] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:04.985176 30714 raft_consensus.cc:374] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:04.985227 30714 raft_consensus.cc:729] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e9b43ceabb479c9c2ab9a131babca8, State: Initialized, Role: FOLLOWER
I20260928 02:46:04.985307 30712 ts_tablet_manager.cc:1428] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:46:04.985644 30561 heartbeater.cc:503] Master 127.28.248.190:46205 was elected leader, sending a full tablet report...
I20260928 02:46:04.985648 30714 consensus_queue.cc:260] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:04.985955 30713 raft_consensus.cc:348] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:04.986199 30713 raft_consensus.cc:374] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:04.986245 30713 raft_consensus.cc:729] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b1512597db42c79e075923a4c3c6ef, State: Initialized, Role: FOLLOWER
I20260928 02:46:04.986253 30427 heartbeater.cc:503] Master 127.28.248.190:46205 was elected leader, sending a full tablet report...
I20260928 02:46:04.986558 30714 ts_tablet_manager.cc:1428] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:46:04.986644 30713 consensus_queue.cc:260] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:04.987351 30695 heartbeater.cc:503] Master 127.28.248.190:46205 was elected leader, sending a full tablet report...
I20260928 02:46:04.987641 30713 ts_tablet_manager.cc:1428] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:46:05.008934 30719 raft_consensus.cc:482] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:05.009164 30719 raft_consensus.cc:504] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:05.010335 30719 leader_election.cc:288] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01474a4664c84905aa6cad537ecbe4f8 (127.28.248.130:40751), 07b1512597db42c79e075923a4c3c6ef (127.28.248.131:44063)
I20260928 02:46:05.014187 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.010668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:44861 (local address 127.28.248.130:40751)
0928 02:46:05.010817 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:05.010822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:05.010976 (+ 154us) server_negotiation.cc:407] Connection header received
0928 02:46:05.011142 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:05.011145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:05.011188 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:05.011272 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:05.011550 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.012048 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.013182 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.013586 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.013724 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.013776 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:05.013973 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:46:05.014045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:46:05.014194 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.010778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:36589 (local address 127.28.248.131:44063)
0928 02:46:05.010898 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:05.010903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:05.011080 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:46:05.011249 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:05.011251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:05.011290 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:05.011364 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:05.011660 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.012133 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.013222 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.013621 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.013754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.013802 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:05.013980 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:46:05.014053 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:46:05.014256 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.010605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:44063 (local address 127.28.248.129:36589)
0928 02:46:05.010986 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:05.010997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:05.011114 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:05.011385 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:05.011388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:05.011403 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:05.011554 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:05.011559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.012239 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.012244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:05.013099 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:05.013108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.013745 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.013748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:05.013858 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.013875 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:46:05.013965 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:46:05.014077 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":65,"thread_start_us":97,"threads_started":1}
I20260928 02:46:05.014261 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.010524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40751 (local address 127.28.248.129:44861)
0928 02:46:05.010882 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:05.010895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:05.011019 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:05.011281 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:05.011284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:05.011299 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:05.011439 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:05.011446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.012155 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.012160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:05.013060 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:05.013069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.013717 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.013721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:05.013825 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.013853 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:46:05.013943 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:46:05.014042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260928 02:46:05.014894 30649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac886059cf4f9f9fafba5f5ed016a8" candidate_uuid: "05e9b43ceabb479c9c2ab9a131babca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b1512597db42c79e075923a4c3c6ef" is_pre_election: true
I20260928 02:46:05.014936 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac886059cf4f9f9fafba5f5ed016a8" candidate_uuid: "05e9b43ceabb479c9c2ab9a131babca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01474a4664c84905aa6cad537ecbe4f8" is_pre_election: true
I20260928 02:46:05.015187 30649 raft_consensus.cc:2454] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e9b43ceabb479c9c2ab9a131babca8 in term 0.
I20260928 02:46:05.015208 30515 raft_consensus.cc:2454] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e9b43ceabb479c9c2ab9a131babca8 in term 0.
I20260928 02:46:05.015671 30316 leader_election.cc:302] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01474a4664c84905aa6cad537ecbe4f8, 05e9b43ceabb479c9c2ab9a131babca8; no voters:
I20260928 02:46:05.015961 30719 raft_consensus.cc:2781] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:05.016057 30719 raft_consensus.cc:482] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:05.016113 30719 raft_consensus.cc:3037] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:05.017278 30719 raft_consensus.cc:504] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:05.017583 30719 leader_election.cc:288] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [CANDIDATE]: Term 1 election: Requested vote from peers 01474a4664c84905aa6cad537ecbe4f8 (127.28.248.130:40751), 07b1512597db42c79e075923a4c3c6ef (127.28.248.131:44063)
I20260928 02:46:05.017959 30649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac886059cf4f9f9fafba5f5ed016a8" candidate_uuid: "05e9b43ceabb479c9c2ab9a131babca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b1512597db42c79e075923a4c3c6ef"
I20260928 02:46:05.017959 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccac886059cf4f9f9fafba5f5ed016a8" candidate_uuid: "05e9b43ceabb479c9c2ab9a131babca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01474a4664c84905aa6cad537ecbe4f8"
I20260928 02:46:05.018097 30515 raft_consensus.cc:3037] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:05.018097 30649 raft_consensus.cc:3037] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:05.019500 30515 raft_consensus.cc:2454] T ccac886059cf4f9f9fafba5f5ed016a8 P 01474a4664c84905aa6cad537ecbe4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e9b43ceabb479c9c2ab9a131babca8 in term 1.
I20260928 02:46:05.019500 30649 raft_consensus.cc:2454] T ccac886059cf4f9f9fafba5f5ed016a8 P 07b1512597db42c79e075923a4c3c6ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e9b43ceabb479c9c2ab9a131babca8 in term 1.
I20260928 02:46:05.019817 30314 leader_election.cc:302] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e9b43ceabb479c9c2ab9a131babca8, 07b1512597db42c79e075923a4c3c6ef; no voters:
I20260928 02:46:05.020000 30719 raft_consensus.cc:2781] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:05.020267 30719 raft_consensus.cc:686] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [term 1 LEADER]: Becoming Leader. State: Replica: 05e9b43ceabb479c9c2ab9a131babca8, State: Running, Role: LEADER
I20260928 02:46:05.020596 30719 consensus_queue.cc:237] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } }
I20260928 02:46:05.027740 30235 catalog_manager.cc:5495] T ccac886059cf4f9f9fafba5f5ed016a8 P 05e9b43ceabb479c9c2ab9a131babca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e9b43ceabb479c9c2ab9a131babca8 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "05e9b43ceabb479c9c2ab9a131babca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05e9b43ceabb479c9c2ab9a131babca8" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01474a4664c84905aa6cad537ecbe4f8" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b1512597db42c79e075923a4c3c6ef" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44063 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:05.042969 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.033503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.28.248.129:39143)
0928 02:46:05.033659 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:05.033664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:05.033907 (+ 243us) server_negotiation.cc:407] Connection header received
0928 02:46:05.034094 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:05.034097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:05.034151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:05.034246 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:05.035040 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.035554 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.036473 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.036766 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.036883 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.039442 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:05.039458 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:05.039470 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:05.039494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:05.041083 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:05.041561 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:05.041573 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:05.041574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:05.041693 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:05.042007 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:05.042011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:05.042013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:05.042265 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:05.042496 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:05.042666 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:46:05.042795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563565, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563565, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:05.069044 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.060468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.28.248.129:39143)
0928 02:46:05.060646 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:05.060652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:05.060806 (+ 154us) server_negotiation.cc:407] Connection header received
0928 02:46:05.060915 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:05.060917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:05.060964 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:05.061037 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:05.061671 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.062193 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.063085 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.063314 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.063406 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.065886 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:05.065889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:05.065891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:05.065915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:05.067526 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:05.067878 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:05.067880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:05.067881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:05.067952 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:05.068168 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:05.068170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:05.068171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:05.068341 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:05.068456 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:05.068734 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:46:05.068830 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:05.093922 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:05.085934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.28.248.129:39143)
0928 02:46:05.086082 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:05.086086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:05.086231 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:46:05.086344 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:05.086347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:05.086389 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:05.086451 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:05.087065 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.087562 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.088352 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:05.088558 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:05.088681 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:05.090949 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:05.090952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:05.090954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:05.090976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:05.092504 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:05.092917 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:05.092920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:05.092921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:05.092991 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:05.093209 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:05.093211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:05.093212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:05.093377 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:05.093487 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:05.093634 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:46:05.093722 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:46:05.095255 30341 tablet_service.cc:2908] Rejecting scan request for tablet ccac886059cf4f9f9fafba5f5ed016a8: Uninitialized: clean time has not yet been initialized
I20260928 02:46:05.097287 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30298
I20260928 02:46:05.103513 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30432
I20260928 02:46:05.109031 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30566
I20260928 02:46:05.114504 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30204
2026-09-28T02:46:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2364 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30737](info): set up 2 sockets
Sep 28 02:46:05 dist-test-slave-r238 krb5kdc[30737](info): commencing operation
krb5kdc: starting...
W20260928 02:46:06.152678 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:06 dist-test-slave-r238 krb5kdc[30737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:06.280911 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:34655
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:06.394136 30750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:06.394452 30750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:06.394495 30750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:06.398936 30750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:06.399015 30750 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:46:06.399034 30750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:06.399052 30750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:06.399070 30750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:06.399085 30750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:06.404574 30750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:34655
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:34655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:06.405702 30750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:06.406741 30750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:46:06.416051 30750 server_base.cc:1034] running on GCE node
W20260928 02:46:06.416030 30755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.416013 30756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.416258 30758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:06.416945 30750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:06.418167 30750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:06.419370 30750 hybrid_clock.cc:648] HybridClock initialized: now 1790563566419353 us; error 29 us; skew 500 ppm
Sep 28 02:46:06 dist-test-slave-r238 krb5kdc[30737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:06.426234 30750 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:06.427299 30750 webserver.cc:460] Webserver started at http://127.28.248.190:39537/ using document root <none> and password file <none>
I20260928 02:46:06.427917 30750 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:06.428011 30750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:06.428253 30750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:06.430503 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "4e242b071ac745bda343146db3747522"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "9b7004392ef19af474cff0c4ae4d5603"
server_key_iv: "9ad3a015e45961587c3a8bdc9ece88f2"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.431119 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "4e242b071ac745bda343146db3747522"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "9b7004392ef19af474cff0c4ae4d5603"
server_key_iv: "9ad3a015e45961587c3a8bdc9ece88f2"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.435164 30750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:06.437947 30765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:06.439347 30750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:46:06.439527 30750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "4e242b071ac745bda343146db3747522"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "9b7004392ef19af474cff0c4ae4d5603"
server_key_iv: "9ad3a015e45961587c3a8bdc9ece88f2"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.439653 30750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:06.495724 30750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:06.499606 30750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:06.499868 30750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:06.508882 30750 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:34655
I20260928 02:46:06.508901 30827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:34655 every 8 connection(s)
I20260928 02:46:06.509954 30750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:06.512813 30828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:06.518464 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30750
I20260928 02:46:06.518571 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:06.518728 30828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: Bootstrap starting.
I20260928 02:46:06.518867 29666 external_mini_cluster.cc:1428] Setting key b15a2e1304dbb0de5ee5daee84677c29
I20260928 02:46:06.521476 30828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:06.522297 30828 log.cc:824] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:06.524331 30828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: No bootstrap required, opened a new log
Sep 28 02:46:06 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:06.529179 30828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e242b071ac745bda343146db3747522" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34655 } }
I20260928 02:46:06.529470 30828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:06.529554 30828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e242b071ac745bda343146db3747522, State: Initialized, Role: FOLLOWER
I20260928 02:46:06.530113 30828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e242b071ac745bda343146db3747522" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34655 } }
I20260928 02:46:06.530282 30828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:06.530375 30828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:06.530509 30828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:06.531478 30828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e242b071ac745bda343146db3747522" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34655 } }
I20260928 02:46:06.531827 30828 leader_election.cc:302] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e242b071ac745bda343146db3747522; no voters:
I20260928 02:46:06.532176 30828 leader_election.cc:288] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:06.532274 30833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:06.532471 30833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [term 1 LEADER]: Becoming Leader. State: Replica: 4e242b071ac745bda343146db3747522, State: Running, Role: LEADER
I20260928 02:46:06.532850 30833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e242b071ac745bda343146db3747522" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34655 } }
I20260928 02:46:06.533586 30828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:06.534745 30834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e242b071ac745bda343146db3747522. Latest consensus state: current_term: 1 leader_uuid: "4e242b071ac745bda343146db3747522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e242b071ac745bda343146db3747522" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34655 } } }
I20260928 02:46:06.534859 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:06.520401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.28.248.190:34655)
0928 02:46:06.520909 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:06.520936 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:06.520976 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:46:06.521763 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:06.521786 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:06.522172 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:06.522555 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:06.523406 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.524419 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.525673 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.526162 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.526299 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:06.529320 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:06.529325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:06.529341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:06.529373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:06.531336 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.531939 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.531947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.531955 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.532091 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.532401 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.532404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.532406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.533009 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:06.533216 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:06.533534 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:46:06.533864 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":226,"threads_started":1}
I20260928 02:46:06.534881 30834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:06.535436 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e242b071ac745bda343146db3747522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e242b071ac745bda343146db3747522" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34655 } } }
I20260928 02:46:06.535995 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522 [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:06.538972 30849 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:06.539052 30849 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:06.539124 30842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:06.540009 30842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:06.546073 30842 catalog_manager.cc:1347] Generated new cluster ID: eaa4ee9d61bc451d8327a2724a227503
I20260928 02:46:06.546160 30842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:06.560039 30842 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:06.561183 30842 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:06.568194 30842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: Generated new TSK 0
I20260928 02:46:06.569126 30842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:06.605124 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:34655
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:06.722136 30854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:06.722438 30854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:06.722487 30854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:06.726797 30854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:06.726867 30854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:06.726909 30854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:06.726982 30854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:06.732522 30854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:34655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:06.733650 30854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:06.734829 30854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:06.745254 30862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.745283 30860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.745255 30859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:06.745537 30854 server_base.cc:1034] running on GCE node
I20260928 02:46:06.745982 30854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:06.746734 30854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:06.747910 30854 hybrid_clock.cc:648] HybridClock initialized: now 1790563566747897 us; error 27 us; skew 500 ppm
Sep 28 02:46:06 dist-test-slave-r238 krb5kdc[30737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:06.751008 30854 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:06.752058 30854 webserver.cc:460] Webserver started at http://127.28.248.129:40553/ using document root <none> and password file <none>
I20260928 02:46:06.752737 30854 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:06.752836 30854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:06.753087 30854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:06.755334 30854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "8f5f9b944acf4adba1160f7db6e24ead"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "c272269c79d5bd4cf409b98bdcee6156"
server_key_iv: "bc54128227897226509e3ed8a42a01d1"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.755957 30854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "8f5f9b944acf4adba1160f7db6e24ead"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "c272269c79d5bd4cf409b98bdcee6156"
server_key_iv: "bc54128227897226509e3ed8a42a01d1"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.759995 30854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:06.762853 30869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:06.765036 30854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:06.765170 30854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "8f5f9b944acf4adba1160f7db6e24ead"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "c272269c79d5bd4cf409b98bdcee6156"
server_key_iv: "bc54128227897226509e3ed8a42a01d1"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.765358 30854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:06.789314 30854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:06.793300 30854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:06.793517 30854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:06.795102 30854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:06.795154 30854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:06.795238 30854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:06.795275 30854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:46:06 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:06.807569 30854 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:34991
I20260928 02:46:06.807662 30984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:34991 every 8 connection(s)
I20260928 02:46:06.808905 30854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:06.811287 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:06.796168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:33905 (local address 127.28.248.190:34655)
0928 02:46:06.796301 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:06.796308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:06.797512 (+ 1204us) server_negotiation.cc:407] Connection header received
0928 02:46:06.798656 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:06.798660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:06.798752 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:06.798839 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:06.800754 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.801780 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.803369 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.803628 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.803744 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:06.806910 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:06.806914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:06.806917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:06.806951 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:06.808788 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.809393 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.809397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.809398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.809489 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.809902 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.809906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.809908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.810135 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:06.810225 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:06.810752 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 02:46:06.811033 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:06.812083 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:06.796845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.28.248.129:33905)
0928 02:46:06.797361 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:06.797406 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:06.798391 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:06.799321 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:06.799330 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:06.799791 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:06.800516 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:06.800558 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.801975 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.801980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:06.803233 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:06.803243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.803796 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.803801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:06.803986 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:06.804697 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:06.804713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:06.806710 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:06.808922 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:06.808927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:06.808939 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:06.809260 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:06.809590 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:06.809593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:06.809595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:06.809767 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:06.810229 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:06.810234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:06.810473 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 02:46:06.810623 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:46:06.811041 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":149,"threads_started":1}
I20260928 02:46:06.812538 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30854
I20260928 02:46:06.812713 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:06.813053 29666 external_mini_cluster.cc:1428] Setting key e8580cb653ff9766de2393a1f6c44b7c
I20260928 02:46:06.813889 30985 heartbeater.cc:347] Connected to a master server at 127.28.248.190:34655
I20260928 02:46:06.814072 30985 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:06.814646 30985 heartbeater.cc:511] Master 127.28.248.190:34655 requested a full tablet report, sending...
I20260928 02:46:06.816262 30781 ts_manager.cc:194] Registered new tserver with Master: 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129:34991)
I20260928 02:46:06.817827 30781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:33905
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:06.825925 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:06.816859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:43519 (local address 127.28.248.190:34655)
0928 02:46:06.817009 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:06.817016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:06.817399 (+ 383us) server_negotiation.cc:407] Connection header received
0928 02:46:06.817497 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:06.817503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:06.817569 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:06.817667 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:06.818175 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.819122 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.820203 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.820709 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.820853 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:06.822270 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:06.822277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:06.822283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:06.822318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:06.824173 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.824681 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.824686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.824688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.824778 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.825081 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.825084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.825086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.825281 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:06.825392 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:06.825684 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:46:06.825790 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":42}
I20260928 02:46:06.825954 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:06.816809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.28.248.129:43519)
0928 02:46:06.817290 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:06.817313 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:06.817410 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:06.817696 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:06.817699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:06.817927 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:06.818048 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:06.818055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.819267 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.819270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:06.820069 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:06.820079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.820890 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.820893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:06.820976 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:06.821604 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:06.821638 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:06.822096 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:06.824306 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:06.824309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:06.824310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:06.824521 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:06.824886 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:06.824889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:06.824891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:06.824977 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:06.825381 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:06.825384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:06.825513 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 02:46:06.825622 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:46:06.825807 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":290,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:06.844012 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:34655
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:06.960337 30994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:06.960592 30994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:06.960657 30994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:06.964946 30994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:06.965008 30994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:06.965049 30994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:06.965081 30994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:06.970810 30994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:34655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:06.971871 30994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:06.972919 30994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:06.982802 31000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.982795 30999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.982795 31002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:06.982869 30994 server_base.cc:1034] running on GCE node
I20260928 02:46:06.983593 30994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:06.984308 30994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:06.985518 30994 hybrid_clock.cc:648] HybridClock initialized: now 1790563566985491 us; error 46 us; skew 500 ppm
Sep 28 02:46:06 dist-test-slave-r238 krb5kdc[30737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:06.988570 30994 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:06.989570 30994 webserver.cc:460] Webserver started at http://127.28.248.130:46475/ using document root <none> and password file <none>
I20260928 02:46:06.990145 30994 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:06.990260 30994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:06.990492 30994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:06.992529 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "bc2129b5b37a4d4d8613a358663da1ed"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "815b12336e93d14fd141b62a28e20d72"
server_key_iv: "a5626bd9a1d9d7284f36fb5abbc71f3f"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.993172 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "bc2129b5b37a4d4d8613a358663da1ed"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "815b12336e93d14fd141b62a28e20d72"
server_key_iv: "a5626bd9a1d9d7284f36fb5abbc71f3f"
server_key_version: "encryptionkey@0"
I20260928 02:46:06.996923 30994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:06.999440 31009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.000699 30994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:07.000829 30994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "bc2129b5b37a4d4d8613a358663da1ed"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-r238"
server_key: "815b12336e93d14fd141b62a28e20d72"
server_key_iv: "a5626bd9a1d9d7284f36fb5abbc71f3f"
server_key_version: "encryptionkey@0"
I20260928 02:46:07.000944 30994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:07.018615 30994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:07.022176 30994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:07.022387 30994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:07.024132 30994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:07.024191 30994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.024258 30994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:07.024289 30994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.037613 30994 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:36407
Sep 28 02:46:07 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:07.037698 31124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:36407 every 8 connection(s)
I20260928 02:46:07.038681 30994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:07.041070 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 30994
I20260928 02:46:07.041194 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:07.041442 29666 external_mini_cluster.cc:1428] Setting key ab71381944b9fb65fb6b9c0002c82758
I20260928 02:46:07.042196 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.025053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:52229 (local address 127.28.248.190:34655)
0928 02:46:07.025314 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.025321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.026425 (+ 1104us) server_negotiation.cc:407] Connection header received
0928 02:46:07.027640 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.027645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.027717 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.027819 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.029468 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.030309 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.034736 (+ 4427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.035115 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.035238 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.038309 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.038312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.038314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.038338 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.039935 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.040476 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.040478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.040479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.040558 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.041007 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.041009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.041010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.041175 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.041289 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.041901 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.042018 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260928 02:46:07.042929 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.025545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.28.248.130:52229)
0928 02:46:07.026214 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.026324 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.027360 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.028146 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.028157 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.028696 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:07.029305 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.029323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.030485 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.030492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.034596 (+ 4104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.034609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.035273 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.035276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.035458 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.036265 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:07.036287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:07.038134 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:07.040037 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.040041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.040052 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.040378 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.040695 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.040698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.040699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.040883 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.041345 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:07.041350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:07.041600 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.041844 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.042139 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":134,"threads_started":1}
I20260928 02:46:07.045405 31125 heartbeater.cc:347] Connected to a master server at 127.28.248.190:34655
I20260928 02:46:07.045536 31125 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:07.045954 31125 heartbeater.cc:511] Master 127.28.248.190:34655 requested a full tablet report, sending...
I20260928 02:46:07.046885 30782 ts_manager.cc:194] Registered new tserver with Master: bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407)
I20260928 02:46:07.047513 30782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:52229
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:46:07.056883 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.047648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.28.248.130:42161)
0928 02:46:07.048839 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.048857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.048999 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.049282 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.049288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.049520 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:07.049679 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.049690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.050481 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.050487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.051458 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.051473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.051961 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.051968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.052076 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.052580 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:07.052600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:07.053056 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:07.054984 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.054987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.054988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.055261 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.055645 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.055650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.055653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.055802 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.056284 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:07.056288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:07.056393 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.056499 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.056673 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":88,"threads_started":1}
I20260928 02:46:07.056919 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.047530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:42161 (local address 127.28.248.190:34655)
0928 02:46:07.047650 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.047654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.048928 (+ 1274us) server_negotiation.cc:407] Connection header received
0928 02:46:07.049112 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.049115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.049158 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.049242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.049827 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.050331 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.051608 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.051836 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.051938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.053252 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.053256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.053259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.053291 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.054882 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.055380 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.055383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.055384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.055458 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.055954 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.055956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.055957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.056165 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.056283 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.056561 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.056779 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:07.073047 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:34655
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:07.191663 31134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:07.191988 31134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:07.192075 31134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:07.196218 31134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:07.196321 31134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:07.196400 31134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:07.196496 31134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:07.202371 31134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:34655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:07.203500 31134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:07.204593 31134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:07.214488 31140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:07.214509 31139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:07.214520 31142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:07.214773 31134 server_base.cc:1034] running on GCE node
I20260928 02:46:07.215378 31134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:07.216025 31134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:07.217223 31134 hybrid_clock.cc:648] HybridClock initialized: now 1790563567217204 us; error 29 us; skew 500 ppm
Sep 28 02:46:07 dist-test-slave-r238 krb5kdc[30737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:07.219933 31134 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:07.220887 31134 webserver.cc:460] Webserver started at http://127.28.248.131:38331/ using document root <none> and password file <none>
I20260928 02:46:07.221467 31134 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:07.221557 31134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:07.221793 31134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:07.223868 31134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "ee2503d85b524371a804ad941e0d8fa1"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-r238"
server_key: "68d3b7acab9ea96d4cd236b095ebf6c8"
server_key_iv: "3aeb15f6cc47abc45e470ab5c84b0f53"
server_key_version: "encryptionkey@0"
I20260928 02:46:07.224455 31134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "ee2503d85b524371a804ad941e0d8fa1"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-r238"
server_key: "68d3b7acab9ea96d4cd236b095ebf6c8"
server_key_iv: "3aeb15f6cc47abc45e470ab5c84b0f53"
server_key_version: "encryptionkey@0"
I20260928 02:46:07.228207 31134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:07.230703 31149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.232081 31134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:07.232250 31134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "ee2503d85b524371a804ad941e0d8fa1"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-r238"
server_key: "68d3b7acab9ea96d4cd236b095ebf6c8"
server_key_iv: "3aeb15f6cc47abc45e470ab5c84b0f53"
server_key_version: "encryptionkey@0"
I20260928 02:46:07.232370 31134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:07.253237 31134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:07.256796 31134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:07.257012 31134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:07.258443 31134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:07.258491 31134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.258531 31134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:07.258587 31134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.268731 31134 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:41339
I20260928 02:46:07.268847 31264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:41339 every 8 connection(s)
Sep 28 02:46:07 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:07.269713 31134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:07.273706 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.259563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:59009 (local address 127.28.248.190:34655)
0928 02:46:07.259700 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.259704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.260857 (+ 1153us) server_negotiation.cc:407] Connection header received
0928 02:46:07.262347 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.262352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.262417 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.262499 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.264142 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.265054 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.266504 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.266869 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.267001 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.269969 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.269972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.269975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.270005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.271597 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.272111 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.272116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.272118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.272210 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.272541 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.272546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.272548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.272798 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.272901 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.273397 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.273522 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:07.274459 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.260113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.28.248.131:59009)
0928 02:46:07.260699 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.260759 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.261933 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.262702 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.262709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.263226 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:07.263954 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.263975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.265189 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.265194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.266377 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.266386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.267050 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.267058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.267312 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.267911 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:07.267929 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:07.269787 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:07.271736 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.271742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.271755 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.272010 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.272304 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.272308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.272310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.272448 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.272909 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:07.272917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:07.273150 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.273334 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.273645 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":149,"threads_started":1}
I20260928 02:46:07.275607 31265 heartbeater.cc:347] Connected to a master server at 127.28.248.190:34655
I20260928 02:46:07.275764 31265 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:07.276259 31265 heartbeater.cc:511] Master 127.28.248.190:34655 requested a full tablet report, sending...
I20260928 02:46:07.277410 30781 ts_manager.cc:194] Registered new tserver with Master: ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:07.278159 30781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:59009
I20260928 02:46:07.279709 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 31134
I20260928 02:46:07.279825 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:07.280095 29666 external_mini_cluster.cc:1428] Setting key 42f99d8681b4834766f81c9abfc1dce2
I20260928 02:46:07.282526 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:07.285465 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.277767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.28.248.131:44325)
0928 02:46:07.278040 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.278054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.278149 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.278412 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.278416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.278630 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:07.278750 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.278757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.279745 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.279750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.280536 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.280545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.281120 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.281124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.281248 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.281783 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:07.281797 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:07.282143 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:07.283831 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.283837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.283842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.284123 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.284451 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.284457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.284460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.284552 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.285068 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:07.285070 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:07.285142 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.285223 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.285330 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260928 02:46:07.285629 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.277851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:44325 (local address 127.28.248.190:34655)
0928 02:46:07.277973 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.277977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.278123 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:46:07.278230 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.278232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.278287 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.278366 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.278870 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.279609 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.280678 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.280894 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.281003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.282332 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.282335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.282337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.282357 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.283714 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.284223 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.284225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.284226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.284339 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.284671 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.284675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.284678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.284965 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.285078 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.285308 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.285448 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 02:46:07 dist-test-slave-r238 krb5kdc[30737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:46:07 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:07.309168 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.301150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.28.248.190:34655)
0928 02:46:07.301289 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.301292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.301306 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:07.301346 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.301349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.301388 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.301459 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.302119 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.302630 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.303408 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.303614 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.303692 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.306155 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.306158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.306160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.306186 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.307689 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.308042 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.308044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.308045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.308113 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.308332 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.308334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.308335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.308491 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.308584 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.308852 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.308941 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:46:07.312134 30781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:07.315308 30781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:07.329946 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.323447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:36407 (local address 127.0.0.1:37766)
0928 02:46:07.323829 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.323884 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.324012 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.324954 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.324959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.324978 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.325143 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.325152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.326812 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.326816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.328214 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.328225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.329415 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.329419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.329521 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.329551 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.329623 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.329765 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260928 02:46:07.330281 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.324152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41339 (local address 127.0.0.1:56788)
0928 02:46:07.324840 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.324852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.324963 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.325787 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.325790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.325821 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.326010 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.326016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.327503 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.327507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.328699 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.328708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.329891 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.329895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.329989 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.330016 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.330088 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.330159 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":111,"spinlock_wait_cycles":2816,"thread_start_us":89,"threads_started":1}
I20260928 02:46:07.330296 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.324681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.28.248.129:34991)
0928 02:46:07.325050 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.325058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.325239 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:46:07.325343 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.325348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.325526 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.325663 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.326140 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.327301 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.328554 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.329210 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.329602 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.329744 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.329796 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.329902 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":68,"threads_started":1}
I20260928 02:46:07.330914 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.323592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.28.248.130:36407)
0928 02:46:07.324407 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.324421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.324446 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:46:07.324525 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.324531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.324779 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.324944 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.325281 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.326678 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.328353 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.329259 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.330512 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.330636 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.330684 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.330776 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":93,"threads_started":1}
I20260928 02:46:07.329949 31284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.324273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34991 (local address 127.0.0.1:55298)
0928 02:46:07.325136 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.325148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.325240 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.325787 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.325790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.325829 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.326008 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.326016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.327435 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.327442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.328426 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.328437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.329352 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.329356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.329478 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.329537 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.329638 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.329777 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":129,"spinlock_wait_cycles":14720,"thread_start_us":81,"threads_started":1}
I20260928 02:46:07.331063 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.324314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.28.248.131:41339)
0928 02:46:07.325254 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.325267 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.325285 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:46:07.325353 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.325357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.325526 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.325663 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.326140 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.327394 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.328823 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.329564 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.329692 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.329831 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.330826 (+ 995us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.330918 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":124,"threads_started":1}
I20260928 02:46:07.333638 31199 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1449797edb49d5817fe48ee8046e8b (DEFAULT_TABLE table=test-table [id=77c9bc9e3be34f8cae142454d19cd25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:07.333636 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1449797edb49d5817fe48ee8046e8b (DEFAULT_TABLE table=test-table [id=77c9bc9e3be34f8cae142454d19cd25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:07.333636 31043 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1449797edb49d5817fe48ee8046e8b (DEFAULT_TABLE table=test-table [id=77c9bc9e3be34f8cae142454d19cd25e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:07.334756 31199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1449797edb49d5817fe48ee8046e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.334749 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1449797edb49d5817fe48ee8046e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.334753 31043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1449797edb49d5817fe48ee8046e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.341559 31288 tablet_bootstrap.cc:492] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Bootstrap starting.
I20260928 02:46:07.342700 31289 tablet_bootstrap.cc:492] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Bootstrap starting.
I20260928 02:46:07.343143 31290 tablet_bootstrap.cc:492] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Bootstrap starting.
I20260928 02:46:07.344507 31288 tablet_bootstrap.cc:654] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.345505 31288 log.cc:824] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:07.346431 31290 tablet_bootstrap.cc:654] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.347002 31289 tablet_bootstrap.cc:654] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.347287 31290 log.cc:824] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:07.347579 31288 tablet_bootstrap.cc:492] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: No bootstrap required, opened a new log
I20260928 02:46:07.347798 31289 log.cc:824] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:07.347808 31288 ts_tablet_manager.cc:1397] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:46:07.349334 31290 tablet_bootstrap.cc:492] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: No bootstrap required, opened a new log
I20260928 02:46:07.349586 31290 ts_tablet_manager.cc:1397] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:46:07.349710 31289 tablet_bootstrap.cc:492] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: No bootstrap required, opened a new log
I20260928 02:46:07.349886 31289 ts_tablet_manager.cc:1397] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 02:46:07.351133 31288 raft_consensus.cc:348] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.351464 31288 raft_consensus.cc:374] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.351528 31288 raft_consensus.cc:729] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2129b5b37a4d4d8613a358663da1ed, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.351984 31288 consensus_queue.cc:260] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.353011 31288 ts_tablet_manager.cc:1428] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:46:07.353375 31125 heartbeater.cc:503] Master 127.28.248.190:34655 was elected leader, sending a full tablet report...
I20260928 02:46:07.354385 31289 raft_consensus.cc:348] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.354651 31289 raft_consensus.cc:374] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.354717 31289 raft_consensus.cc:729] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5f9b944acf4adba1160f7db6e24ead, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.355019 31290 raft_consensus.cc:348] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.355159 31289 consensus_queue.cc:260] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.355265 31290 raft_consensus.cc:374] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.355316 31290 raft_consensus.cc:729] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2503d85b524371a804ad941e0d8fa1, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.355779 31290 consensus_queue.cc:260] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.355861 30985 heartbeater.cc:503] Master 127.28.248.190:34655 was elected leader, sending a full tablet report...
I20260928 02:46:07.356165 31289 ts_tablet_manager.cc:1428] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 02:46:07.356902 31265 heartbeater.cc:503] Master 127.28.248.190:34655 was elected leader, sending a full tablet report...
I20260928 02:46:07.356753 31290 ts_tablet_manager.cc:1428] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:46:07.422147 31295 raft_consensus.cc:482] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:07.422387 31295 raft_consensus.cc:504] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.423528 31295 leader_election.cc:288] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407), ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:07.426469 31296 raft_consensus.cc:482] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:07.426651 31296 raft_consensus.cc:504] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.427963 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.423877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:41169 (local address 127.28.248.130:36407)
0928 02:46:07.424023 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.424027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.424188 (+ 161us) server_negotiation.cc:407] Connection header received
0928 02:46:07.424532 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.424537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.424595 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.424710 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.425035 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.425638 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.426926 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.427352 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.427477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.427532 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.427709 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.427786 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:07.428157 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.424381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:60581 (local address 127.28.248.131:41339)
0928 02:46:07.424560 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.424563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.424684 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:46:07.424832 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.424835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.424892 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.424981 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.425265 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.425805 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.427034 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.427570 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.427690 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.427735 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.427955 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.428040 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:46:07.428426 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.423774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:36407 (local address 127.28.248.129:41169)
0928 02:46:07.424080 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.424096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.424215 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.424735 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.424739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.424765 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.424919 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.424931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.425744 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.425747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.426820 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.426830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.427468 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.427471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.427567 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.427589 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.428162 (+ 573us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.428243 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":132,"threads_started":1}
I20260928 02:46:07.428774 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.424229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41339 (local address 127.28.248.129:60581)
0928 02:46:07.424585 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.424599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.424726 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.425017 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.425020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.425036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.425147 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.425153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.425911 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.425913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.426914 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.426923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.427690 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.427694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.427789 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.427810 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.428549 (+ 739us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.428646 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":108,"threads_started":1}
I20260928 02:46:07.429121 31078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1449797edb49d5817fe48ee8046e8b" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2129b5b37a4d4d8613a358663da1ed" is_pre_election: true
I20260928 02:46:07.429447 31078 raft_consensus.cc:2454] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 0.
I20260928 02:46:07.429622 31296 leader_election.cc:288] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407), 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129:34991)
I20260928 02:46:07.429970 30872 leader_election.cc:302] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5f9b944acf4adba1160f7db6e24ead, bc2129b5b37a4d4d8613a358663da1ed; no voters:
I20260928 02:46:07.430086 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1449797edb49d5817fe48ee8046e8b" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2503d85b524371a804ad941e0d8fa1" is_pre_election: true
I20260928 02:46:07.430259 31219 raft_consensus.cc:2454] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 0.
I20260928 02:46:07.430264 31295 raft_consensus.cc:2781] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:07.430337 31295 raft_consensus.cc:482] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:07.430380 31295 raft_consensus.cc:3037] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:07.431550 31295 raft_consensus.cc:504] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.431927 31295 leader_election.cc:288] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 election: Requested vote from peers bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407), ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:07.432240 31078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1449797edb49d5817fe48ee8046e8b" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2129b5b37a4d4d8613a358663da1ed"
I20260928 02:46:07.432276 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1449797edb49d5817fe48ee8046e8b" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2503d85b524371a804ad941e0d8fa1"
I20260928 02:46:07.432361 31078 raft_consensus.cc:3037] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:07.432385 31219 raft_consensus.cc:3037] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:07.432747 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.428553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:40913 (local address 127.28.248.130:36407)
0928 02:46:07.428712 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.428716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.428729 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:07.428771 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.428775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.428835 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.428913 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.429159 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.429809 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.431362 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.431955 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.432461 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.432502 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.432533 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.432593 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:46:07.433739 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.428384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:60595 (local address 127.28.248.129:34991)
0928 02:46:07.428505 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.428509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.429415 (+ 906us) server_negotiation.cc:407] Connection header received
0928 02:46:07.429499 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.429502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.429547 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.429621 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.430685 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.431382 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.432752 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.433208 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.433282 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.433314 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.433581 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.433642 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:46:07.433763 31219 raft_consensus.cc:2454] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 1.
I20260928 02:46:07.433768 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.428086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34991 (local address 127.28.248.131:60595)
0928 02:46:07.429309 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.429334 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.429423 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.429906 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.429912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.429934 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.430554 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.430564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.431499 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.431502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.432600 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.432635 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.433396 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.433398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.433466 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.433480 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.433567 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.433628 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":45,"thread_start_us":98,"threads_started":1}
I20260928 02:46:07.434122 31078 raft_consensus.cc:2454] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 1.
I20260928 02:46:07.434408 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.428059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:36407 (local address 127.28.248.131:40913)
0928 02:46:07.428285 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.428298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.428385 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.428938 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.428941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.428957 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.429055 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.429060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.429933 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.429936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.431229 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.431238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.432083 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.432087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.432193 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.432238 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.432284 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.432346 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":22}
I20260928 02:46:07.434432 30872 leader_election.cc:302] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5f9b944acf4adba1160f7db6e24ead, ee2503d85b524371a804ad941e0d8fa1; no voters:
I20260928 02:46:07.434577 31295 raft_consensus.cc:2781] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:07.434774 31078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1449797edb49d5817fe48ee8046e8b" candidate_uuid: "ee2503d85b524371a804ad941e0d8fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2129b5b37a4d4d8613a358663da1ed" is_pre_election: true
I20260928 02:46:07.434975 31078 raft_consensus.cc:2379] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee2503d85b524371a804ad941e0d8fa1 in current term 1: Already voted for candidate 8f5f9b944acf4adba1160f7db6e24ead in this term.
I20260928 02:46:07.434942 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1449797edb49d5817fe48ee8046e8b" candidate_uuid: "ee2503d85b524371a804ad941e0d8fa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5f9b944acf4adba1160f7db6e24ead" is_pre_election: true
I20260928 02:46:07.435422 31295 raft_consensus.cc:686] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 1 LEADER]: Becoming Leader. State: Replica: 8f5f9b944acf4adba1160f7db6e24ead, State: Running, Role: LEADER
I20260928 02:46:07.435999 31295 consensus_queue.cc:237] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } }
I20260928 02:46:07.438659 31153 leader_election.cc:302] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee2503d85b524371a804ad941e0d8fa1; no voters: 8f5f9b944acf4adba1160f7db6e24ead, bc2129b5b37a4d4d8613a358663da1ed
I20260928 02:46:07.439142 31296 raft_consensus.cc:2726] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:46:07.439683 30781 catalog_manager.cc:5495] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129). New cstate: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } health_report { overall_health: HEALTHY } } }
I20260928 02:46:07.456925 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.453521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.28.248.129:34991)
0928 02:46:07.453641 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.453645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.453658 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:07.453859 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.453862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.453922 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.454011 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:07.454254 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.454746 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.455560 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.455761 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.455834 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.456103 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:07.456513 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:07.456598 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.456757 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.456804 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:46:07.462424 31078 raft_consensus.cc:1261] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Refusing update from remote peer 8f5f9b944acf4adba1160f7db6e24ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:07.462471 31219 raft_consensus.cc:1261] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Refusing update from remote peer 8f5f9b944acf4adba1160f7db6e24ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:07.463011 31295 consensus_queue.cc:1035] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:07.463249 31300 consensus_queue.cc:1035] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:07.470022 31303 mvcc.cc:205] Tried to move back new op lower bound from 7334148372319424512 to 7334148372221091840. Current Snapshot: MvccSnapshot[applied={T|T < 7334148372319424512}]
I20260928 02:46:07.482507 30781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:07.482672 30781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:07.485224 30781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: Sending DeleteTablet for 3 replicas of tablet 6a1449797edb49d5817fe48ee8046e8b
I20260928 02:46:07.486109 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1449797edb49d5817fe48ee8046e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:07 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55298
I20260928 02:46:07.486132 31199 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1449797edb49d5817fe48ee8046e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:07 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:56788
I20260928 02:46:07.486343 31043 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1449797edb49d5817fe48ee8046e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:07 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:37766
I20260928 02:46:07.486637 31313 tablet_replica.cc:330] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: stopping tablet replica
I20260928 02:46:07.486855 31315 tablet_replica.cc:330] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: stopping tablet replica
I20260928 02:46:07.487002 31313 raft_consensus.cc:2229] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:07.487253 31315 raft_consensus.cc:2229] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:07.487347 31313 raft_consensus.cc:2258] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:07.487550 30781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:46:07.487664 31315 raft_consensus.cc:2258] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:46:07.488005 30781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:07.488322 31314 tablet_replica.cc:330] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: stopping tablet replica
I20260928 02:46:07.488710 31314 raft_consensus.cc:2229] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:07.489110 31314 raft_consensus.cc:2258] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:07.489110 31313 ts_tablet_manager.cc:1905] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:07.489708 31315 ts_tablet_manager.cc:1905] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:07.493364 31315 ts_tablet_manager.cc:1918] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:07.493464 31315 log.cc:1196] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/6a1449797edb49d5817fe48ee8046e8b
I20260928 02:46:07.493675 31313 ts_tablet_manager.cc:1918] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:07.493767 31313 log.cc:1196] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/6a1449797edb49d5817fe48ee8046e8b
I20260928 02:46:07.493768 31315 ts_tablet_manager.cc:1939] T 6a1449797edb49d5817fe48ee8046e8b P bc2129b5b37a4d4d8613a358663da1ed: Deleting consensus metadata
I20260928 02:46:07.494102 31313 ts_tablet_manager.cc:1939] T 6a1449797edb49d5817fe48ee8046e8b P ee2503d85b524371a804ad941e0d8fa1: Deleting consensus metadata
I20260928 02:46:07.494104 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 20a0b72430274fe3b687e72af4366072 (DEFAULT_TABLE table=test-table [id=919e4fa5c9b84e71ac77fc97edbea827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:07.494385 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a0b72430274fe3b687e72af4366072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.495153 30768 catalog_manager.cc:4841] TS bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407): tablet 6a1449797edb49d5817fe48ee8046e8b (table test-table [id=77c9bc9e3be34f8cae142454d19cd25e]) successfully deleted
I20260928 02:46:07.495523 31199 tablet_service.cc:1467] Processing CreateTablet for tablet 20a0b72430274fe3b687e72af4366072 (DEFAULT_TABLE table=test-table [id=919e4fa5c9b84e71ac77fc97edbea827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:07.495743 31199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a0b72430274fe3b687e72af4366072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.496369 31289 tablet_bootstrap.cc:492] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Bootstrap starting.
I20260928 02:46:07.496762 30768 catalog_manager.cc:4841] TS ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339): tablet 6a1449797edb49d5817fe48ee8046e8b (table test-table [id=77c9bc9e3be34f8cae142454d19cd25e]) successfully deleted
I20260928 02:46:07.497179 31043 tablet_service.cc:1467] Processing CreateTablet for tablet 20a0b72430274fe3b687e72af4366072 (DEFAULT_TABLE table=test-table [id=919e4fa5c9b84e71ac77fc97edbea827]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:07.497424 31289 tablet_bootstrap.cc:654] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.497466 31043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a0b72430274fe3b687e72af4366072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.498047 31290 tablet_bootstrap.cc:492] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Bootstrap starting.
I20260928 02:46:07.498677 31314 ts_tablet_manager.cc:1905] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:07.499221 31290 tablet_bootstrap.cc:654] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.500047 31288 tablet_bootstrap.cc:492] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Bootstrap starting.
I20260928 02:46:07.500483 31289 tablet_bootstrap.cc:492] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: No bootstrap required, opened a new log
I20260928 02:46:07.500577 31289 ts_tablet_manager.cc:1397] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:46:07.501009 31288 tablet_bootstrap.cc:654] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.501116 31289 raft_consensus.cc:348] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.501261 31289 raft_consensus.cc:374] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.501300 31289 raft_consensus.cc:729] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5f9b944acf4adba1160f7db6e24ead, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.501431 31289 consensus_queue.cc:260] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.501720 31289 ts_tablet_manager.cc:1428] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:07.502079 31314 ts_tablet_manager.cc:1918] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:07.502218 31314 log.cc:1196] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/6a1449797edb49d5817fe48ee8046e8b
I20260928 02:46:07.502596 31314 ts_tablet_manager.cc:1939] T 6a1449797edb49d5817fe48ee8046e8b P 8f5f9b944acf4adba1160f7db6e24ead: Deleting consensus metadata
I20260928 02:46:07.503571 31288 tablet_bootstrap.cc:492] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: No bootstrap required, opened a new log
I20260928 02:46:07.503690 31288 ts_tablet_manager.cc:1397] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:46:07.503823 30769 catalog_manager.cc:4841] TS 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129:34991): tablet 6a1449797edb49d5817fe48ee8046e8b (table test-table [id=77c9bc9e3be34f8cae142454d19cd25e]) successfully deleted
I20260928 02:46:07.504185 31288 raft_consensus.cc:348] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.504302 31288 raft_consensus.cc:374] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.504344 31288 raft_consensus.cc:729] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2129b5b37a4d4d8613a358663da1ed, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.504490 31288 consensus_queue.cc:260] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.504666 31290 tablet_bootstrap.cc:492] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: No bootstrap required, opened a new log
I20260928 02:46:07.504745 31290 ts_tablet_manager.cc:1397] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 02:46:07.504784 31288 ts_tablet_manager.cc:1428] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:07.505167 31290 raft_consensus.cc:348] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.505265 31290 raft_consensus.cc:374] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.505294 31290 raft_consensus.cc:729] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2503d85b524371a804ad941e0d8fa1, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.505415 31290 consensus_queue.cc:260] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.505646 31290 ts_tablet_manager.cc:1428] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 02:46:07.520787 31266 tablet.cc:2369] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:46:07.539908 31126 tablet.cc:2369] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:46:07.560300 30986 tablet.cc:2369] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:07.861572 31294 raft_consensus.cc:482] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:07.861763 31294 raft_consensus.cc:504] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.862926 31294 leader_election.cc:288] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129:34991), ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:07.867084 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.863222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:53229 (local address 127.28.248.129:34991)
0928 02:46:07.863474 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.863479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.863499 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:46:07.863688 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.863691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.863754 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.863813 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.864189 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.864748 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.865981 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.866483 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.866584 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.866626 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.866855 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.866939 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 02:46:07.867112 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.863119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34991 (local address 127.28.248.130:53229)
0928 02:46:07.863415 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.863432 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.863535 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.863879 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.863882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.863898 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.864063 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.864068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.864859 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.864865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.865838 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.865848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.866608 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.866612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.866720 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.866752 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.866852 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.866931 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":134,"thread_start_us":80,"threads_started":1}
I20260928 02:46:07.867224 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.863301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:56109 (local address 127.28.248.131:41339)
0928 02:46:07.863430 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.863435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.863577 (+ 142us) server_negotiation.cc:407] Connection header received
0928 02:46:07.863738 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.863740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.863779 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.863873 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:07.864173 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.864753 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.866223 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.866663 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.866779 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.866840 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.867033 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.867099 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:46:07.867425 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.863155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41339 (local address 127.28.248.130:56109)
0928 02:46:07.863476 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.863491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.863600 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.863916 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.863920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.863936 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:07.864063 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.864068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.864883 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.864893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.866097 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.866107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.866783 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.866786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.866903 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.866925 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.867000 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.867062 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":22,"thread_start_us":79,"threads_started":1}
I20260928 02:46:07.867679 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a0b72430274fe3b687e72af4366072" candidate_uuid: "bc2129b5b37a4d4d8613a358663da1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5f9b944acf4adba1160f7db6e24ead" is_pre_election: true
I20260928 02:46:07.867875 30932 raft_consensus.cc:2454] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2129b5b37a4d4d8613a358663da1ed in term 0.
I20260928 02:46:07.867887 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a0b72430274fe3b687e72af4366072" candidate_uuid: "bc2129b5b37a4d4d8613a358663da1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2503d85b524371a804ad941e0d8fa1" is_pre_election: true
I20260928 02:46:07.868044 31219 raft_consensus.cc:2454] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc2129b5b37a4d4d8613a358663da1ed in term 0.
I20260928 02:46:07.868250 31013 leader_election.cc:302] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5f9b944acf4adba1160f7db6e24ead, bc2129b5b37a4d4d8613a358663da1ed; no voters:
I20260928 02:46:07.868587 31294 raft_consensus.cc:2781] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:07.868692 31294 raft_consensus.cc:482] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:07.868765 31294 raft_consensus.cc:3037] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:07.869899 31294 raft_consensus.cc:504] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.870316 31294 leader_election.cc:288] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [CANDIDATE]: Term 1 election: Requested vote from peers 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129:34991), ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:07.870612 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a0b72430274fe3b687e72af4366072" candidate_uuid: "bc2129b5b37a4d4d8613a358663da1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5f9b944acf4adba1160f7db6e24ead"
I20260928 02:46:07.870663 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a0b72430274fe3b687e72af4366072" candidate_uuid: "bc2129b5b37a4d4d8613a358663da1ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2503d85b524371a804ad941e0d8fa1"
I20260928 02:46:07.870769 30932 raft_consensus.cc:3037] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:07.870784 31219 raft_consensus.cc:3037] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:07.871702 31219 raft_consensus.cc:2454] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2129b5b37a4d4d8613a358663da1ed in term 1.
I20260928 02:46:07.871779 30932 raft_consensus.cc:2454] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2129b5b37a4d4d8613a358663da1ed in term 1.
I20260928 02:46:07.872001 31012 leader_election.cc:302] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc2129b5b37a4d4d8613a358663da1ed, ee2503d85b524371a804ad941e0d8fa1; no voters:
I20260928 02:46:07.872229 31294 raft_consensus.cc:2781] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:07.872373 31294 raft_consensus.cc:686] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 1 LEADER]: Becoming Leader. State: Replica: bc2129b5b37a4d4d8613a358663da1ed, State: Running, Role: LEADER
I20260928 02:46:07.872670 31294 consensus_queue.cc:237] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.874783 30781 catalog_manager.cc:5495] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed reported cstate change: term changed from 0 to 1, leader changed from <none> to bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130). New cstate: current_term: 1 leader_uuid: "bc2129b5b37a4d4d8613a358663da1ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:46:07 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:07.979241 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.970486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.28.248.190:34655)
0928 02:46:07.970766 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.970774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.970961 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:46:07.971076 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.971078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.971120 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.971193 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.971885 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.972470 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.973345 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.973660 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.973742 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.976092 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.976095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.976097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.976115 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.977456 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.977922 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.977926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.977928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.978006 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.978274 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.978278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.978280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.978461 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.978565 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.978855 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.979034 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260928 02:46:07.979242 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.970445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34655 (local address 127.0.0.1:42754)
0928 02:46:07.970869 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.970882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.970979 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.971229 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.971231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.971639 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:07.971734 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.971740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.972575 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.972577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.973237 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.973245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.973759 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.973761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.973819 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.974424 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:07.974435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:07.975954 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:07.977561 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.977573 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.977579 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.977806 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.978101 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.978104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.978106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.978187 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.978573 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:07.978583 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:07.978694 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.978811 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.979089 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":111,"thread_start_us":105,"threads_started":1}
I20260928 02:46:07.983132 30781 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:42754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:46:07.983893 30781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:07.988727 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2c245687ce41938134de1589925abb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b11a7e01ba14597a63d6e593852d0a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:07.988760 31043 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2c245687ce41938134de1589925abb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b11a7e01ba14597a63d6e593852d0a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:07.988855 31199 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2c245687ce41938134de1589925abb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b11a7e01ba14597a63d6e593852d0a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:07.989141 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2c245687ce41938134de1589925abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.989151 31043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2c245687ce41938134de1589925abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.989149 31199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2c245687ce41938134de1589925abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:07.991641 31289 tablet_bootstrap.cc:492] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: Bootstrap starting.
I20260928 02:46:07.991992 31290 tablet_bootstrap.cc:492] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: Bootstrap starting.
I20260928 02:46:07.992161 31288 tablet_bootstrap.cc:492] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: Bootstrap starting.
I20260928 02:46:07.992488 31289 tablet_bootstrap.cc:654] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.992959 31290 tablet_bootstrap.cc:654] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.993075 31288 tablet_bootstrap.cc:654] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:07.993973 31289 tablet_bootstrap.cc:492] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: No bootstrap required, opened a new log
I20260928 02:46:07.994062 31289 ts_tablet_manager.cc:1397] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:07.994091 31290 tablet_bootstrap.cc:492] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: No bootstrap required, opened a new log
I20260928 02:46:07.994153 31290 ts_tablet_manager.cc:1397] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:07.994411 31288 tablet_bootstrap.cc:492] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: No bootstrap required, opened a new log
I20260928 02:46:07.994513 31288 ts_tablet_manager.cc:1397] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:07.994475 31289 raft_consensus.cc:348] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.994604 31289 raft_consensus.cc:374] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.994575 31290 raft_consensus.cc:348] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.994647 31289 raft_consensus.cc:729] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5f9b944acf4adba1160f7db6e24ead, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.994689 31290 raft_consensus.cc:374] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.994730 31290 raft_consensus.cc:729] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2503d85b524371a804ad941e0d8fa1, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.994778 31289 consensus_queue.cc:260] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.994835 31290 consensus_queue.cc:260] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.995066 31289 ts_tablet_manager.cc:1428] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:07.994994 31288 raft_consensus.cc:348] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.995111 31290 ts_tablet_manager.cc:1428] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:07.995123 31288 raft_consensus.cc:374] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:07.995154 31288 raft_consensus.cc:729] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2129b5b37a4d4d8613a358663da1ed, State: Initialized, Role: FOLLOWER
I20260928 02:46:07.995273 31288 consensus_queue.cc:260] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:07.995513 31288 ts_tablet_manager.cc:1428] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:07.995548 31296 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:07.995671 31296 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:07.995698 31300 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:07.995822 31300 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:07.995517 31294 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:07.996002 31294 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:08.099282 31300 raft_consensus.cc:482] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:08.099483 31300 raft_consensus.cc:504] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:08.099975 31300 leader_election.cc:288] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407), ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:08.100538 31078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2c245687ce41938134de1589925abb" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2129b5b37a4d4d8613a358663da1ed" is_pre_election: true
I20260928 02:46:08.100546 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2c245687ce41938134de1589925abb" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2503d85b524371a804ad941e0d8fa1" is_pre_election: true
I20260928 02:46:08.100718 31078 raft_consensus.cc:2454] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 0.
I20260928 02:46:08.100729 31219 raft_consensus.cc:2454] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 0.
I20260928 02:46:08.101055 30872 leader_election.cc:302] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5f9b944acf4adba1160f7db6e24ead, ee2503d85b524371a804ad941e0d8fa1; no voters:
I20260928 02:46:08.101269 31300 raft_consensus.cc:2781] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:08.101343 31300 raft_consensus.cc:482] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:08.101397 31300 raft_consensus.cc:3037] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:08.102430 31300 raft_consensus.cc:504] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:08.102758 31300 leader_election.cc:288] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 election: Requested vote from peers bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407), ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339)
I20260928 02:46:08.103152 31078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2c245687ce41938134de1589925abb" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc2129b5b37a4d4d8613a358663da1ed"
I20260928 02:46:08.103291 31078 raft_consensus.cc:3037] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:08.103277 31219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2c245687ce41938134de1589925abb" candidate_uuid: "8f5f9b944acf4adba1160f7db6e24ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2503d85b524371a804ad941e0d8fa1"
I20260928 02:46:08.103427 31219 raft_consensus.cc:3037] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:08.104444 31219 raft_consensus.cc:2454] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 1.
I20260928 02:46:08.104458 31078 raft_consensus.cc:2454] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f5f9b944acf4adba1160f7db6e24ead in term 1.
I20260928 02:46:08.104768 30872 leader_election.cc:302] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5f9b944acf4adba1160f7db6e24ead, ee2503d85b524371a804ad941e0d8fa1; no voters:
I20260928 02:46:08.104990 31300 raft_consensus.cc:2781] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:08.105183 31300 raft_consensus.cc:686] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [term 1 LEADER]: Becoming Leader. State: Replica: 8f5f9b944acf4adba1160f7db6e24ead, State: Running, Role: LEADER
I20260928 02:46:08.105342 31300 consensus_queue.cc:237] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } }
I20260928 02:46:08.106012 31333 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: TxnStatusTablet state changed. Reason: New leader 8f5f9b944acf4adba1160f7db6e24ead. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.106139 31333 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:08.106383 31335 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:46:08.106575 31335 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:46:08.107285 30781 catalog_manager.cc:5495] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129). New cstate: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:08.119763 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.116449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34991 (local address 127.0.0.1:55312)
0928 02:46:08.116624 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.116637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.116756 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.117122 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.117125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.117132 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:08.117237 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.117242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.117970 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.117975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.118579 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.118586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.119037 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.119041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.119127 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.119167 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.119466 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.119481 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.119568 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.119649 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:08.119763 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.116533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.28.248.129:34991)
0928 02:46:08.116922 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.116925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.116936 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:08.116970 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.116972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.117009 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.117095 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:08.117344 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.117844 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.118696 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.118925 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.119036 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.119303 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.119379 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.119460 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.119582 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.119633 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260928 02:46:08.122503 31219 raft_consensus.cc:1261] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Refusing update from remote peer 8f5f9b944acf4adba1160f7db6e24ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:08.122503 31078 raft_consensus.cc:1261] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Refusing update from remote peer 8f5f9b944acf4adba1160f7db6e24ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:08.122937 31333 consensus_queue.cc:1035] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:08.123099 31300 consensus_queue.cc:1035] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:08.124504 31294 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: TxnStatusTablet state changed. Reason: New leader 8f5f9b944acf4adba1160f7db6e24ead. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.124648 31294 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:08.124547 31296 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: TxnStatusTablet state changed. Reason: New leader 8f5f9b944acf4adba1160f7db6e24ead. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.124687 31296 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:08.126582 31300 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.126719 31300 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:08.127919 31296 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.128018 31296 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P ee2503d85b524371a804ad941e0d8fa1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:08.128273 31333 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.128391 31333 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P 8f5f9b944acf4adba1160f7db6e24ead: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:08.128657 31340 mvcc.cc:205] Tried to move back new op lower bound from 7334148375025651712 to 7334148374960914432. Current Snapshot: MvccSnapshot[applied={T|T < 7334148375025651712}]
I20260928 02:46:08.129190 31294 tablet_replica.cc:439] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f5f9b944acf4adba1160f7db6e24ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 } } peers { permanent_uuid: "bc2129b5b37a4d4d8613a358663da1ed" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36407 } } peers { permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 } } }
I20260928 02:46:08.129302 31294 tablet_replica.cc:442] T 3f2c245687ce41938134de1589925abb P bc2129b5b37a4d4d8613a358663da1ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:08.132164 31339 mvcc.cc:205] Tried to move back new op lower bound from 7334148375025651712 to 7334148374960914432. Current Snapshot: MvccSnapshot[applied={T|T < 7334148375025651712}]
I20260928 02:46:08.137557 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.132506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.28.248.130:36407)
0928 02:46:08.133026 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.133029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.133067 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:46:08.133141 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.133144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.133210 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.133320 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:08.133646 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.134515 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.135534 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.135922 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.136034 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.136249 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.136912 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.137315 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.137360 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.137413 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":62,"threads_started":1}
I20260928 02:46:08.147917 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.144426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:58249 (local address 127.28.248.129:34991)
0928 02:46:08.144560 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.144567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.144826 (+ 259us) server_negotiation.cc:407] Connection header received
0928 02:46:08.144985 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.144988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.145043 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.145124 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:08.145392 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.145958 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.146820 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.147040 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.147149 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.147458 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.147529 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.147606 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.147701 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.147754 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:46:08.148444 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.144328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34991 (local address 127.28.248.130:58249)
0928 02:46:08.144740 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.144752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.144868 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.145134 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.145137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.145144 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:08.145282 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.145288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.146063 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.146068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.146697 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.146705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.147151 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.147155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.147258 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.147301 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.147621 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.147629 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.147691 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.147771 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":87,"threads_started":1}
I20260928 02:46:08.155844 31219 raft_consensus.cc:1261] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Refusing update from remote peer bc2129b5b37a4d4d8613a358663da1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:08.155968 30932 raft_consensus.cc:1261] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Refusing update from remote peer bc2129b5b37a4d4d8613a358663da1ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:08.156347 31294 consensus_queue.cc:1035] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2503d85b524371a804ad941e0d8fa1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:08.156535 31321 consensus_queue.cc:1035] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f5f9b944acf4adba1160f7db6e24ead" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:08.178565 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.175022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:49165 (local address 127.28.248.130:36407)
0928 02:46:08.175162 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.175167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.175180 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:08.175281 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.175284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.175332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.175427 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:08.175713 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.176296 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.177489 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.177724 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.177839 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.178125 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.178201 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.178281 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.178368 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.178421 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:46:08.178572 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.174751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:36407 (local address 127.28.248.129:49165)
0928 02:46:08.175080 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.175098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.175197 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.175436 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.175439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.175446 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:08.175573 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.175578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.176412 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.176415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.177363 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.177380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.177838 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.177842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.177938 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.177990 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:08.178286 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:08.178295 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.178358 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.178443 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260928 02:46:08.206782 30782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:08.206964 30782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:08.209287 30782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e242b071ac745bda343146db3747522: Sending DeleteTablet for 3 replicas of tablet 20a0b72430274fe3b687e72af4366072
I20260928 02:46:08.209921 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a0b72430274fe3b687e72af4366072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:08 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55298
I20260928 02:46:08.209944 31043 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a0b72430274fe3b687e72af4366072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:08 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:37766
I20260928 02:46:08.210562 31199 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a0b72430274fe3b687e72af4366072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:08 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:56788
I20260928 02:46:08.210640 31360 tablet_replica.cc:330] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: stopping tablet replica
I20260928 02:46:08.210758 31360 raft_consensus.cc:2229] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:08.210860 31360 raft_consensus.cc:2258] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:08.211537 31360 ts_tablet_manager.cc:1905] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:08.212037 31362 tablet_replica.cc:330] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: stopping tablet replica
I20260928 02:46:08.212697 31362 raft_consensus.cc:2229] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:08.213248 31362 raft_consensus.cc:2258] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:08.213971 31362 ts_tablet_manager.cc:1905] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:08.214485 31360 ts_tablet_manager.cc:1918] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:08.214607 31360 log.cc:1196] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/20a0b72430274fe3b687e72af4366072
I20260928 02:46:08.214965 31360 ts_tablet_manager.cc:1939] T 20a0b72430274fe3b687e72af4366072 P 8f5f9b944acf4adba1160f7db6e24ead: Deleting consensus metadata
I20260928 02:46:08.215688 30769 catalog_manager.cc:4841] TS 8f5f9b944acf4adba1160f7db6e24ead (127.28.248.129:34991): tablet 20a0b72430274fe3b687e72af4366072 (table test-table [id=919e4fa5c9b84e71ac77fc97edbea827]) successfully deleted
I20260928 02:46:08.216492 31361 tablet_replica.cc:330] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: stopping tablet replica
I20260928 02:46:08.216737 31361 raft_consensus.cc:2229] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:08.216912 31362 ts_tablet_manager.cc:1918] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:08.217044 31362 log.cc:1196] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/20a0b72430274fe3b687e72af4366072
I20260928 02:46:08.217126 31361 raft_consensus.cc:2258] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:08.217358 31362 ts_tablet_manager.cc:1939] T 20a0b72430274fe3b687e72af4366072 P ee2503d85b524371a804ad941e0d8fa1: Deleting consensus metadata
I20260928 02:46:08.217815 31361 ts_tablet_manager.cc:1905] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:08.218056 30768 catalog_manager.cc:4841] TS ee2503d85b524371a804ad941e0d8fa1 (127.28.248.131:41339): tablet 20a0b72430274fe3b687e72af4366072 (table test-table [id=919e4fa5c9b84e71ac77fc97edbea827]) successfully deleted
I20260928 02:46:08.220521 31361 ts_tablet_manager.cc:1918] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:08.220600 31361 log.cc:1196] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/20a0b72430274fe3b687e72af4366072
I20260928 02:46:08.220943 31361 ts_tablet_manager.cc:1939] T 20a0b72430274fe3b687e72af4366072 P bc2129b5b37a4d4d8613a358663da1ed: Deleting consensus metadata
I20260928 02:46:08.221647 30768 catalog_manager.cc:4841] TS bc2129b5b37a4d4d8613a358663da1ed (127.28.248.130:36407): tablet 20a0b72430274fe3b687e72af4366072 (table test-table [id=919e4fa5c9b84e71ac77fc97edbea827]) successfully deleted
Sep 28 02:46:08 dist-test-slave-r238 krb5kdc[30737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:08.233188 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.212456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.28.248.129:34991)
0928 02:46:08.212563 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.212568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.220955 (+ 8387us) server_negotiation.cc:407] Connection header received
0928 02:46:08.221124 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.221127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.221177 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.221248 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.222208 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.223003 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.224391 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.224779 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.224889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.228918 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.228930 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.228948 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.228978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.231136 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.231638 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.231645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.231647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.231754 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.232061 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.232065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.232067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.232403 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:08.232716 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.232872 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.233008 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 02:46:08.233626 30979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55326
I20260928 02:46:08.246518 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.235609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.28.248.190:34655)
0928 02:46:08.235742 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.235746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.236902 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 02:46:08.237035 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.237038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.237097 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.237184 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.238045 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.238845 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.240084 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.240464 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.240583 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.242337 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.242341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.242344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.242372 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.244401 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.244980 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.244983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.244985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.245125 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.245459 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.245462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.245464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.245738 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:08.245920 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.246211 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.246349 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:46:08.247020 30782 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42758
I20260928 02:46:08.248580 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30854
I20260928 02:46:08.257198 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30994
I20260928 02:46:08.262892 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 31134
I20260928 02:46:08.268709 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 30750
2026-09-28T02:46:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3154 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:08 dist-test-slave-r238 krb5kdc[31369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:08 dist-test-slave-r238 krb5kdc[31369](info): set up 2 sockets
Sep 28 02:46:08 dist-test-slave-r238 krb5kdc[31369](info): commencing operation
krb5kdc: starting...
W20260928 02:46:09.316797 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:09 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:09.435642 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39719
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:09.544073 31382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:09.544338 31382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:09.544377 31382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:09.548393 31382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:09.548455 31382 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:46:09.548472 31382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:09.548491 31382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:09.548506 31382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:09.548521 31382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:09.553745 31382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:39719
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:09.554792 31382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:09.555840 31382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:09.564443 31388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:09.564448 31387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:09.564640 31382 server_base.cc:1034] running on GCE node
W20260928 02:46:09.564458 31390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:09.565299 31382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:09.566543 31382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:09.567737 31382 hybrid_clock.cc:648] HybridClock initialized: now 1790563569567729 us; error 32 us; skew 500 ppm
Sep 28 02:46:09 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:09.570802 31382 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:09.571681 31382 webserver.cc:460] Webserver started at http://127.28.248.190:44233/ using document root <none> and password file <none>
I20260928 02:46:09.572243 31382 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:09.572302 31382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:09.572460 31382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:09.574527 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "6fb74a9318934ee6b7fd311093df5c3c"
format_stamp: "Formatted at 2026-09-28 02:46:09 on dist-test-slave-r238"
server_key: "f4f0018261a0b84502c4b6287c5f5e50"
server_key_iv: "6d0befcb54b6c66cd673f63bcec6b9d9"
server_key_version: "encryptionkey@0"
I20260928 02:46:09.575029 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "6fb74a9318934ee6b7fd311093df5c3c"
format_stamp: "Formatted at 2026-09-28 02:46:09 on dist-test-slave-r238"
server_key: "f4f0018261a0b84502c4b6287c5f5e50"
server_key_iv: "6d0befcb54b6c66cd673f63bcec6b9d9"
server_key_version: "encryptionkey@0"
I20260928 02:46:09.578917 31382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:09.581264 31397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:09.582468 31382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:09.582563 31382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "6fb74a9318934ee6b7fd311093df5c3c"
format_stamp: "Formatted at 2026-09-28 02:46:09 on dist-test-slave-r238"
server_key: "f4f0018261a0b84502c4b6287c5f5e50"
server_key_iv: "6d0befcb54b6c66cd673f63bcec6b9d9"
server_key_version: "encryptionkey@0"
I20260928 02:46:09.582648 31382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:09.598654 31382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:09.601791 31382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:09.601959 31382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:09.610402 31382 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:39719
I20260928 02:46:09.610421 31459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:39719 every 8 connection(s)
I20260928 02:46:09.611439 31382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:09.611795 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 31382
I20260928 02:46:09.611910 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:09.612224 29666 external_mini_cluster.cc:1428] Setting key deda2ba84b8a926f28ee9c025675747a
I20260928 02:46:09.614959 31460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:09.621775 31460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Bootstrap starting.
Sep 28 02:46:09 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:09.624661 31460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:09.626055 31460 log.cc:824] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:09.627640 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:09.613717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.28.248.190:39719)
0928 02:46:09.614129 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:09.614148 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:09.614189 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:46:09.615040 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:09.615063 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:09.615477 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:09.615838 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:09.616579 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.618134 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.619098 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.619771 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.619968 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:09.622524 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:09.622529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:09.622546 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:09.622576 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:09.624477 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.625033 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.625041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.625048 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.625217 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.625482 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.625485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.625486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.626014 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:09.626210 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:09.626573 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 02:46:09.626918 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260928 02:46:09.628381 31460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: No bootstrap required, opened a new log
I20260928 02:46:09.631316 31460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb74a9318934ee6b7fd311093df5c3c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39719 } }
I20260928 02:46:09.631553 31460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:09.631646 31460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb74a9318934ee6b7fd311093df5c3c, State: Initialized, Role: FOLLOWER
I20260928 02:46:09.632054 31460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb74a9318934ee6b7fd311093df5c3c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39719 } }
I20260928 02:46:09.632200 31460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:09.632292 31460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:09.632400 31460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:09.633323 31460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb74a9318934ee6b7fd311093df5c3c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39719 } }
I20260928 02:46:09.633603 31460 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb74a9318934ee6b7fd311093df5c3c; no voters:
I20260928 02:46:09.633863 31460 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:09.633988 31465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:09.634214 31465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [term 1 LEADER]: Becoming Leader. State: Replica: 6fb74a9318934ee6b7fd311093df5c3c, State: Running, Role: LEADER
I20260928 02:46:09.634555 31465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb74a9318934ee6b7fd311093df5c3c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39719 } }
I20260928 02:46:09.635114 31460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:09.636307 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb74a9318934ee6b7fd311093df5c3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb74a9318934ee6b7fd311093df5c3c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39719 } } }
I20260928 02:46:09.636309 31467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb74a9318934ee6b7fd311093df5c3c. Latest consensus state: current_term: 1 leader_uuid: "6fb74a9318934ee6b7fd311093df5c3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb74a9318934ee6b7fd311093df5c3c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39719 } } }
I20260928 02:46:09.636458 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:09.636471 31467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:09.636824 31474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:09.639935 31474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:09.645604 31474 catalog_manager.cc:1347] Generated new cluster ID: a93a600b4b7d403193402c16452aae7f
I20260928 02:46:09.645670 31474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:09.654959 31474 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:09.656169 31474 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:09.668455 31474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Generated new TSK 0
I20260928 02:46:09.669410 31474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:09.707830 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39719
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:09.822386 31486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:09.822662 31486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:09.822711 31486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:09.826741 31486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:09.826797 31486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:09.826874 31486 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:46:09.826908 31486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:09.826998 31486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:09.832183 31486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:09.833266 31486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:09.834278 31486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:09.843796 31492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:09.843799 31494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:09.843809 31491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:09.844044 31486 server_base.cc:1034] running on GCE node
I20260928 02:46:09.844476 31486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:09.845139 31486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:09.846336 31486 hybrid_clock.cc:648] HybridClock initialized: now 1790563569846322 us; error 35 us; skew 500 ppm
Sep 28 02:46:09 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:09.849224 31486 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:09.850039 31486 webserver.cc:460] Webserver started at http://127.28.248.129:40453/ using document root <none> and password file <none>
I20260928 02:46:09.850561 31486 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:09.850618 31486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:09.850776 31486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:09.852810 31486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
format_stamp: "Formatted at 2026-09-28 02:46:09 on dist-test-slave-r238"
server_key: "61a7e5b76fde380a7256f3aa0a80990e"
server_key_iv: "38f81d737c57b294ced1b645f8b17248"
server_key_version: "encryptionkey@0"
I20260928 02:46:09.853384 31486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
format_stamp: "Formatted at 2026-09-28 02:46:09 on dist-test-slave-r238"
server_key: "61a7e5b76fde380a7256f3aa0a80990e"
server_key_iv: "38f81d737c57b294ced1b645f8b17248"
server_key_version: "encryptionkey@0"
I20260928 02:46:09.857213 31486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:46:09.859472 31501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:09.860754 31486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:09.860878 31486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
format_stamp: "Formatted at 2026-09-28 02:46:09 on dist-test-slave-r238"
server_key: "61a7e5b76fde380a7256f3aa0a80990e"
server_key_iv: "38f81d737c57b294ced1b645f8b17248"
server_key_version: "encryptionkey@0"
I20260928 02:46:09.860966 31486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:09.872305 31486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:09.875763 31486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:09.875927 31486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:09.877426 31486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:09.877502 31486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:09.877592 31486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:09.877638 31486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:46:09 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:09.888376 31486 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:45677
I20260928 02:46:09.888499 31616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:45677 every 8 connection(s)
I20260928 02:46:09.889434 31486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:09.891578 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:09.878085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:43033 (local address 127.28.248.190:39719)
0928 02:46:09.878246 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:09.878253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:09.879557 (+ 1304us) server_negotiation.cc:407] Connection header received
0928 02:46:09.880835 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:09.880838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:09.880898 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:09.880979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:09.882271 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.882987 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.884469 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.884888 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.884997 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:09.887925 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:09.887928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:09.887930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:09.887955 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:09.889569 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.890206 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.890212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.890213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.890306 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.890677 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.890680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.890681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.890855 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:09.890932 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:09.891360 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 02:46:09.891464 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:09.892344 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:09.878953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.28.248.129:43033)
0928 02:46:09.879394 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:09.879444 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:09.880594 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:09.881154 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:09.881161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:09.881637 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:09.882122 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:09.882136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.883119 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.883123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:09.884318 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:09.884332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.885041 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.885045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:09.885241 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:09.885927 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:09.885943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:09.887757 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:09.889703 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:09.889709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:09.889722 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:09.890078 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:09.890428 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:09.890431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:09.890433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:09.890591 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:09.890940 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:09.890944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:09.891155 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 02:46:09.891305 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:46:09.891549 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260928 02:46:09.893604 31617 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39719
I20260928 02:46:09.893741 31617 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:09.894387 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 31486
I20260928 02:46:09.894379 31617 heartbeater.cc:511] Master 127.28.248.190:39719 requested a full tablet report, sending...
I20260928 02:46:09.894522 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:09.894861 29666 external_mini_cluster.cc:1428] Setting key 4b8dcf9d45f41220587cd98020aab324
I20260928 02:46:09.896070 31413 ts_manager.cc:194] Registered new tserver with Master: 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677)
I20260928 02:46:09.897717 31413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:43033
I20260928 02:46:09.903781 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:09.895991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:49103 (local address 127.28.248.190:39719)
0928 02:46:09.896809 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:09.896812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:09.896823 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:09.896851 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:09.896853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:09.896885 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:09.896973 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:09.897414 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.898145 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.899308 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.899531 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.899635 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:09.900760 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:09.900764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:09.900766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:09.900789 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:09.902194 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.902624 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.902628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.902630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.902705 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.902983 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.902986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.902988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.903153 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:09.903263 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:09.903506 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:46:09.903615 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":705}
I20260928 02:46:09.903797 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:09.896113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.28.248.129:49103)
0928 02:46:09.896347 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:09.896361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:09.896442 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:09.896990 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:09.896993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:09.897199 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:09.897304 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:09.897309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.898307 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.898311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:09.899187 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:09.899196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:09.899638 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:09.899641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:09.899733 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:09.900241 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:09.900253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:09.900563 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:09.902321 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:09.902327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:09.902329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:09.902528 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:09.902804 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:09.902807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:09.902809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:09.902890 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:09.903246 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:09.903248 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:09.903338 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:46:09.903444 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:46:09.903581 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:09.923972 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39719
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:10.035146 31626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:10.035394 31626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:10.035466 31626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:10.039724 31626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:10.039779 31626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:10.039882 31626 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:46:10.039913 31626 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:10.039984 31626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:10.045094 31626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:10.046094 31626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:10.047088 31626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:10.056377 31631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:10.056489 31626 server_base.cc:1034] running on GCE node
W20260928 02:46:10.056432 31632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:10.056408 31634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:10.056975 31626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:10.057591 31626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:10.058804 31626 hybrid_clock.cc:648] HybridClock initialized: now 1790563570058785 us; error 27 us; skew 500 ppm
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:10.061551 31626 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:10.062366 31626 webserver.cc:460] Webserver started at http://127.28.248.130:41641/ using document root <none> and password file <none>
I20260928 02:46:10.062850 31626 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:10.062906 31626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:10.063056 31626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:10.065066 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "c843cabb00a84a889da46ace27bc5717"
format_stamp: "Formatted at 2026-09-28 02:46:10 on dist-test-slave-r238"
server_key: "942abae329ad317435c9562dda7a0c55"
server_key_iv: "20fb477dc083334ca567485586968c64"
server_key_version: "encryptionkey@0"
I20260928 02:46:10.065577 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "c843cabb00a84a889da46ace27bc5717"
format_stamp: "Formatted at 2026-09-28 02:46:10 on dist-test-slave-r238"
server_key: "942abae329ad317435c9562dda7a0c55"
server_key_iv: "20fb477dc083334ca567485586968c64"
server_key_version: "encryptionkey@0"
I20260928 02:46:10.069115 31626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:10.071599 31641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:10.072846 31626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:10.072958 31626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "c843cabb00a84a889da46ace27bc5717"
format_stamp: "Formatted at 2026-09-28 02:46:10 on dist-test-slave-r238"
server_key: "942abae329ad317435c9562dda7a0c55"
server_key_iv: "20fb477dc083334ca567485586968c64"
server_key_version: "encryptionkey@0"
I20260928 02:46:10.073060 31626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:10.086171 31626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:10.089179 31626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:10.089330 31626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:10.090719 31626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:10.090793 31626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:10.090865 31626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:10.090916 31626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:10.099756 31626 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:33177
I20260928 02:46:10.100015 31756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:33177 every 8 connection(s)
I20260928 02:46:10.100855 31626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:10.105037 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.091471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:48313 (local address 127.28.248.190:39719)
0928 02:46:10.091641 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.091649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.092829 (+ 1180us) server_negotiation.cc:407] Connection header received
0928 02:46:10.094160 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.094166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.094254 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.094359 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.095984 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.096910 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.098160 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.098568 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.098689 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.101721 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.101727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.101732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.101764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.102981 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.103491 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.103496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.103498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.103581 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.103934 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.103939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.103941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.104119 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.104226 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.104738 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.104858 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:46:10.105788 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.092243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.28.248.130:48313)
0928 02:46:10.092693 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.092741 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.093914 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.094553 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.094564 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.095048 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:10.095784 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.095805 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.097044 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.097048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.098014 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.098030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.098744 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.098749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.098972 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.099599 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:10.099617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:10.101542 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:10.103101 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.103106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.103120 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.103372 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.103682 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.103686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.103688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.103828 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.104235 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:10.104244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:10.104481 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.104698 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.104997 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260928 02:46:10.107024 31757 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39719
I20260928 02:46:10.107177 31757 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:10.107663 31757 heartbeater.cc:511] Master 127.28.248.190:39719 requested a full tablet report, sending...
I20260928 02:46:10.108700 31413 ts_manager.cc:194] Registered new tserver with Master: c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177)
I20260928 02:46:10.109445 31413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:48313
I20260928 02:46:10.110432 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 31626
I20260928 02:46:10.110563 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:10.110846 29666 external_mini_cluster.cc:1428] Setting key be0090c903871b5e1fe37c07f050267f
I20260928 02:46:10.116876 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.109229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:47519 (local address 127.28.248.190:39719)
0928 02:46:10.109727 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.109730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.109740 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:46:10.109763 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.109765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.109792 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.109889 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.110335 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.111072 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.112056 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.112301 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.112386 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.113649 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.113652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.113654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.113674 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.115147 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.115626 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.115629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.115630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.115698 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.115992 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.115994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.115995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.116176 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.116298 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.116679 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.116766 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260928 02:46:10.116959 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.109103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.28.248.130:47519)
0928 02:46:10.109395 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.109409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.109495 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.109898 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.109901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.110089 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:10.110208 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.110216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.111209 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.111214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.111940 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.111952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.112457 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.112461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.112580 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.113055 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:10.113074 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:10.113519 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:10.115245 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.115251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.115256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.115530 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.115789 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.115795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.115800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.115901 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.116303 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:10.116307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:10.116448 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.116581 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.116752 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:10.139817 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39719
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:10.249341 31766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:10.249636 31766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:10.249709 31766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:10.253847 31766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:10.253933 31766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:10.254002 31766 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:46:10.254046 31766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:10.254115 31766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:10.259475 31766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:39719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:10.260562 31766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:10.261605 31766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:10.271140 31772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:10.271135 31771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:10.271137 31774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:10.271621 31766 server_base.cc:1034] running on GCE node
I20260928 02:46:10.272065 31766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:10.272807 31766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:10.274021 31766 hybrid_clock.cc:648] HybridClock initialized: now 1790563570274003 us; error 34 us; skew 500 ppm
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:10.277232 31766 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:10.278048 31766 webserver.cc:460] Webserver started at http://127.28.248.131:34089/ using document root <none> and password file <none>
I20260928 02:46:10.278594 31766 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:10.278653 31766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:10.278870 31766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:10.280889 31766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "a2b781616a99441aae68f5849d400daf"
format_stamp: "Formatted at 2026-09-28 02:46:10 on dist-test-slave-r238"
server_key: "be611012c74f74bc7d9ff06e07f18926"
server_key_iv: "b069b5f5486260d5ba646f68296313c0"
server_key_version: "encryptionkey@0"
I20260928 02:46:10.281430 31766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "a2b781616a99441aae68f5849d400daf"
format_stamp: "Formatted at 2026-09-28 02:46:10 on dist-test-slave-r238"
server_key: "be611012c74f74bc7d9ff06e07f18926"
server_key_iv: "b069b5f5486260d5ba646f68296313c0"
server_key_version: "encryptionkey@0"
I20260928 02:46:10.284991 31766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:46:10.287377 31781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:10.288551 31766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:10.288722 31766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "a2b781616a99441aae68f5849d400daf"
format_stamp: "Formatted at 2026-09-28 02:46:10 on dist-test-slave-r238"
server_key: "be611012c74f74bc7d9ff06e07f18926"
server_key_iv: "b069b5f5486260d5ba646f68296313c0"
server_key_version: "encryptionkey@0"
I20260928 02:46:10.288822 31766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:10.302160 31766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:10.305474 31766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:10.305662 31766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:10.306978 31766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:10.307024 31766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:10.307106 31766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:10.307138 31766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563570, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:10.317485 31766 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:41639
I20260928 02:46:10.317555 31896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:41639 every 8 connection(s)
I20260928 02:46:10.318488 31766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:10.321295 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.307905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:50285 (local address 127.28.248.190:39719)
0928 02:46:10.308061 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.308065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.308950 (+ 885us) server_negotiation.cc:407] Connection header received
0928 02:46:10.310092 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.310097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.310169 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.310277 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.311817 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.312688 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.314559 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.314774 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.314858 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.317889 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.317893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.317896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.317923 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.319398 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.319899 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.319906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.319911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.320005 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.320361 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.320363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.320364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.320536 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.320661 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.321088 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.321183 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:46:10.322062 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.308327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.28.248.131:50285)
0928 02:46:10.308780 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.308828 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.309770 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.310488 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.310499 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.310929 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:10.311626 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.311648 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.312853 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.312858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.314418 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.314433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.314952 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.314957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.315187 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.315840 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:10.315866 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:10.317712 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:10.319523 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.319529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.319543 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.319789 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.320105 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.320110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.320114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.320271 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.320662 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:10.320668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:10.320889 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.321041 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.321266 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
I20260928 02:46:10.323228 31897 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39719
I20260928 02:46:10.323359 31897 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:10.323882 31897 heartbeater.cc:511] Master 127.28.248.190:39719 requested a full tablet report, sending...
I20260928 02:46:10.324959 31414 ts_manager.cc:194] Registered new tserver with Master: a2b781616a99441aae68f5849d400daf (127.28.248.131:41639)
I20260928 02:46:10.325595 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 31766
I20260928 02:46:10.325716 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:10.325727 31414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:50285
I20260928 02:46:10.326042 29666 external_mini_cluster.cc:1428] Setting key 944b3a38ed655e9657b5da442ddba30c
I20260928 02:46:10.328789 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:10.333079 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.325516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.28.248.131:46297)
0928 02:46:10.325791 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.325806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.325896 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.326198 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.326202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.326425 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:10.326565 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.326573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.327522 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.327524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.328297 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.328305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.328791 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.328794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.328963 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.329448 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:10.329458 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:10.329740 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:10.331670 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.331673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.331674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.331858 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.332138 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.332139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.332141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.332210 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.332695 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:10.332697 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:10.332767 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.332851 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.332954 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260928 02:46:10.333290 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.325774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:46297 (local address 127.28.248.190:39719)
0928 02:46:10.325901 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.325905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.325920 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:10.325961 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.325964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.326007 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.326086 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.326685 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.327402 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.328406 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.328678 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.328778 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.329904 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.329908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.329911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.329933 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.331563 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.331956 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.331962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.331967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.332048 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.332295 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.332301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.332305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.332547 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.332673 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.332936 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.333062 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:10.354619 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.346859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.28.248.190:39719)
0928 02:46:10.346982 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.346985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.346996 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:10.347031 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.347033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.347068 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.347129 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.347770 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.348272 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.349113 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.349316 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.349406 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.351640 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.351643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.351645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.351666 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.353096 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.353489 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.353492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.353493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.353568 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.353837 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.353841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.353842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.354033 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.354150 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.354348 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.354445 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:10.357673 31414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:10.360767 31414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:10.374854 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.369647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.28.248.130:33177)
0928 02:46:10.370100 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.370114 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.370139 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:46:10.370220 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.370225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.370444 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.370599 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.371043 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.371857 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.373412 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.374143 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.374289 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.374445 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.374595 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.374687 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":122,"threads_started":1}
I20260928 02:46:10.374864 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.369150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33177 (local address 127.0.0.1:57892)
0928 02:46:10.369787 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.369803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.369880 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.370739 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.370742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.370759 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.370876 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.370895 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.371989 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.371995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.373263 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.373273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.374273 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.374279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.374388 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.374426 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.374567 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.374688 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":98,"threads_started":1}
I20260928 02:46:10.374902 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.368956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41639 (local address 127.0.0.1:55380)
0928 02:46:10.369454 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.369504 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.369628 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.370528 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.370533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.370555 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.370723 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.370732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.372470 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.372474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.373707 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.373716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.374484 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.374488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.374582 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.374604 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.374662 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.374734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":51,"thread_start_us":88,"threads_started":1}
I20260928 02:46:10.374918 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.369118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.28.248.131:41639)
0928 02:46:10.369969 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.369983 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.370027 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:46:10.370106 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.370111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.370333 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.370879 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.371036 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.372345 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.373832 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.374359 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.374522 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.374659 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.374704 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.374794 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":128,"threads_started":1}
I20260928 02:46:10.377285 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.369694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:45677 (local address 127.0.0.1:52892)
0928 02:46:10.370245 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.370258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.370335 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.371008 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.371011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.371028 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.371113 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.371119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.372518 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.372522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.373715 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.373724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.376683 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.376687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.376882 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.376903 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.376977 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.377146 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":31,"thread_start_us":87,"threads_started":1}
I20260928 02:46:10.377347 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.370125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.28.248.129:45677)
0928 02:46:10.370590 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.370604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.370624 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:46:10.370694 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.370701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.370872 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.371017 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.371255 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.372384 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.375103 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.375731 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.375880 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.375993 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.377027 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.377204 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":80,"threads_started":1}
I20260928 02:46:10.378216 31824 tablet_service.cc:1467] Processing CreateTablet for tablet 226c406c822e40aba4462891ebe75040 (DEFAULT_TABLE table=test-table [id=c28f3d3356ee46a2b68157dc8cc192bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.378432 31691 tablet_service.cc:1467] Processing CreateTablet for tablet 226c406c822e40aba4462891ebe75040 (DEFAULT_TABLE table=test-table [id=c28f3d3356ee46a2b68157dc8cc192bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.379339 31824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226c406c822e40aba4462891ebe75040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.379364 31691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226c406c822e40aba4462891ebe75040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.380414 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 226c406c822e40aba4462891ebe75040 (DEFAULT_TABLE table=test-table [id=c28f3d3356ee46a2b68157dc8cc192bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.381425 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 226c406c822e40aba4462891ebe75040. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.385910 31920 tablet_bootstrap.cc:492] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Bootstrap starting.
I20260928 02:46:10.387344 31921 tablet_bootstrap.cc:492] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Bootstrap starting.
I20260928 02:46:10.388443 31920 tablet_bootstrap.cc:654] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.389323 31920 log.cc:824] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:10.389467 31921 tablet_bootstrap.cc:654] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.389993 31922 tablet_bootstrap.cc:492] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Bootstrap starting.
I20260928 02:46:10.390288 31921 log.cc:824] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:10.391454 31920 tablet_bootstrap.cc:492] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: No bootstrap required, opened a new log
I20260928 02:46:10.391644 31920 ts_tablet_manager.cc:1397] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:46:10.392231 31922 tablet_bootstrap.cc:654] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.392254 31921 tablet_bootstrap.cc:492] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: No bootstrap required, opened a new log
I20260928 02:46:10.392447 31921 ts_tablet_manager.cc:1397] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:46:10.393282 31922 log.cc:824] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:10.394968 31922 tablet_bootstrap.cc:492] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: No bootstrap required, opened a new log
I20260928 02:46:10.395174 31922 ts_tablet_manager.cc:1397] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:46:10.395653 31920 raft_consensus.cc:348] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.395926 31920 raft_consensus.cc:374] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.395985 31920 raft_consensus.cc:729] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.396304 31921 raft_consensus.cc:348] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.396418 31920 consensus_queue.cc:260] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.396551 31921 raft_consensus.cc:374] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.396624 31921 raft_consensus.cc:729] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b781616a99441aae68f5849d400daf, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.397060 31921 consensus_queue.cc:260] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.397998 31921 ts_tablet_manager.cc:1428] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 02:46:10.397998 31920 ts_tablet_manager.cc:1428] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:46:10.398365 31897 heartbeater.cc:503] Master 127.28.248.190:39719 was elected leader, sending a full tablet report...
I20260928 02:46:10.398532 31757 heartbeater.cc:503] Master 127.28.248.190:39719 was elected leader, sending a full tablet report...
I20260928 02:46:10.398437 31922 raft_consensus.cc:348] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.398651 31922 raft_consensus.cc:374] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.398695 31922 raft_consensus.cc:729] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d03b7b12e48dab2b1507ed71aec8e, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.399084 31922 consensus_queue.cc:260] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.399916 31617 heartbeater.cc:503] Master 127.28.248.190:39719 was elected leader, sending a full tablet report...
I20260928 02:46:10.400120 31922 ts_tablet_manager.cc:1428] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:46:10.495564 31927 raft_consensus.cc:482] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:10.495754 31927 raft_consensus.cc:504] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.496904 31927 leader_election.cc:288] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177), 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677)
I20260928 02:46:10.500584 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.497243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:37339 (local address 127.28.248.130:33177)
0928 02:46:10.497390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.497395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.497411 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:46:10.497553 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.497559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.497622 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.497705 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.498029 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.498575 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.499627 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.500027 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.500156 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.500209 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.500389 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.500462 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:10.500584 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.497124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33177 (local address 127.28.248.131:37339)
0928 02:46:10.497276 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.497311 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.497422 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.497779 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.497782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.497805 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.497915 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.497920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.498697 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.498702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.499499 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.499508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.500146 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.500149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.500256 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.500278 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.500374 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.500448 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:46:10.501340 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.497439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:45677 (local address 127.28.248.131:51231)
0928 02:46:10.497821 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.497831 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.497949 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.498183 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.498186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.498200 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.498674 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.498681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.499401 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.499406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.500156 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.500164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.500950 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.500954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.501057 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.501080 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.501157 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.501217 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260928 02:46:10.501333 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.497569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:51231 (local address 127.28.248.129:45677)
0928 02:46:10.497703 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.497707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.497910 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:46:10.498039 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.498042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.498091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.498173 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.498799 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.499293 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.500285 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.500817 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.500940 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.500986 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.501179 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.501236 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:10.501324 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226c406c822e40aba4462891ebe75040" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843cabb00a84a889da46ace27bc5717" is_pre_election: true
I20260928 02:46:10.501652 31711 raft_consensus.cc:2454] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 0.
I20260928 02:46:10.501938 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226c406c822e40aba4462891ebe75040" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" is_pre_election: true
I20260928 02:46:10.502162 31783 leader_election.cc:302] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b781616a99441aae68f5849d400daf, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:10.502221 31571 raft_consensus.cc:2454] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 0.
I20260928 02:46:10.502471 31927 raft_consensus.cc:2781] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:10.502576 31927 raft_consensus.cc:482] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:10.502645 31927 raft_consensus.cc:3037] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.503697 31927 raft_consensus.cc:504] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.504047 31927 leader_election.cc:288] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 election: Requested vote from peers c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177), 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677)
I20260928 02:46:10.504457 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226c406c822e40aba4462891ebe75040" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843cabb00a84a889da46ace27bc5717"
I20260928 02:46:10.504531 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "226c406c822e40aba4462891ebe75040" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
I20260928 02:46:10.504580 31711 raft_consensus.cc:3037] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.504686 31571 raft_consensus.cc:3037] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.505921 31571 raft_consensus.cc:2454] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 1.
I20260928 02:46:10.505914 31711 raft_consensus.cc:2454] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 1.
I20260928 02:46:10.506218 31783 leader_election.cc:302] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d03b7b12e48dab2b1507ed71aec8e, a2b781616a99441aae68f5849d400daf; no voters:
I20260928 02:46:10.506382 31927 raft_consensus.cc:2781] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:10.506632 31927 raft_consensus.cc:686] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 1 LEADER]: Becoming Leader. State: Replica: a2b781616a99441aae68f5849d400daf, State: Running, Role: LEADER
I20260928 02:46:10.506982 31927 consensus_queue.cc:237] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } }
I20260928 02:46:10.510094 31414 catalog_manager.cc:5495] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b781616a99441aae68f5849d400daf (127.28.248.131). New cstate: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:46:10 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:10.555030 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.546533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39719 (local address 127.0.0.1:50554)
0928 02:46:10.546846 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.546858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.547003 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.547257 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.547260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.547685 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:10.547785 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.547791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.548559 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.548564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.549200 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.549209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.549647 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.549650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.549728 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.550306 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:10.550317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:10.551867 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:10.553550 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.553559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.553564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.553832 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.554081 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:10.554083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:10.554084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:10.554164 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:10.554523 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:10.554534 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:10.554642 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.554736 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.554872 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260928 02:46:10.555051 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.546569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.28.248.190:39719)
0928 02:46:10.546715 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.546720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.546940 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:46:10.547098 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.547102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.547149 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.547236 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.547910 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.548453 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.549321 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.549539 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.549645 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.552009 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.552012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.552014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.552034 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.553430 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.553927 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.553929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.553930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.554000 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.554243 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.554245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.554246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.554412 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.554485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.554792 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.554933 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":48}
I20260928 02:46:10.558739 31414 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:50554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:46:10.559484 31414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:10.564731 31691 tablet_service.cc:1467] Processing CreateTablet for tablet b58cdd8fe631466bbea770499f7cdbce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6a0f63e484c482b95f52fc1d218e537]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:10.564538 31551 tablet_service.cc:1467] Processing CreateTablet for tablet b58cdd8fe631466bbea770499f7cdbce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6a0f63e484c482b95f52fc1d218e537]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:10.565346 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58cdd8fe631466bbea770499f7cdbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.565346 31691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58cdd8fe631466bbea770499f7cdbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.568385 31920 tablet_bootstrap.cc:492] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: Bootstrap starting.
I20260928 02:46:10.568385 31922 tablet_bootstrap.cc:492] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: Bootstrap starting.
I20260928 02:46:10.568828 31824 tablet_service.cc:1467] Processing CreateTablet for tablet b58cdd8fe631466bbea770499f7cdbce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6a0f63e484c482b95f52fc1d218e537]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
W20260928 02:46:10.569388 31898 tablet.cc:2369] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:10.569458 31824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58cdd8fe631466bbea770499f7cdbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.569456 31920 tablet_bootstrap.cc:654] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.569458 31922 tablet_bootstrap.cc:654] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.570930 31922 tablet_bootstrap.cc:492] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: No bootstrap required, opened a new log
I20260928 02:46:10.570995 31920 tablet_bootstrap.cc:492] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: No bootstrap required, opened a new log
I20260928 02:46:10.571026 31922 ts_tablet_manager.cc:1397] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:10.571064 31920 ts_tablet_manager.cc:1397] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:10.571487 31922 raft_consensus.cc:348] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.571487 31920 raft_consensus.cc:348] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.571604 31922 raft_consensus.cc:374] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.571615 31920 raft_consensus.cc:374] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.571645 31922 raft_consensus.cc:729] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d03b7b12e48dab2b1507ed71aec8e, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.571650 31920 raft_consensus.cc:729] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.571790 31920 consensus_queue.cc:260] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.571785 31922 consensus_queue.cc:260] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.572088 31926 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.572234 31926 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.572234 31928 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.572369 31928 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.572590 31922 ts_tablet_manager.cc:1428] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:10.572698 31921 tablet_bootstrap.cc:492] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: Bootstrap starting.
I20260928 02:46:10.572986 31920 ts_tablet_manager.cc:1428] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:10.573593 31921 tablet_bootstrap.cc:654] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.574864 31921 tablet_bootstrap.cc:492] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: No bootstrap required, opened a new log
I20260928 02:46:10.574956 31921 ts_tablet_manager.cc:1397] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:10.575441 31921 raft_consensus.cc:348] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.575564 31921 raft_consensus.cc:374] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.575598 31921 raft_consensus.cc:729] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b781616a99441aae68f5849d400daf, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.575733 31921 consensus_queue.cc:260] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.576053 31921 ts_tablet_manager.cc:1428] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:10.576050 31927 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.576229 31927 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.576898 31927 raft_consensus.cc:482] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:10.576977 31927 raft_consensus.cc:504] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.577330 31927 leader_election.cc:288] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177)
I20260928 02:46:10.577701 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58cdd8fe631466bbea770499f7cdbce" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843cabb00a84a889da46ace27bc5717" is_pre_election: true
I20260928 02:46:10.577832 31711 raft_consensus.cc:2454] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 0.
I20260928 02:46:10.577872 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58cdd8fe631466bbea770499f7cdbce" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" is_pre_election: true
I20260928 02:46:10.578013 31571 raft_consensus.cc:2454] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 0.
I20260928 02:46:10.578118 31783 leader_election.cc:302] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b781616a99441aae68f5849d400daf, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:10.578414 31927 raft_consensus.cc:2781] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:10.578491 31927 raft_consensus.cc:482] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:10.578529 31927 raft_consensus.cc:3037] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.579375 31927 raft_consensus.cc:504] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.579689 31927 leader_election.cc:288] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 election: Requested vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177)
I20260928 02:46:10.580119 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58cdd8fe631466bbea770499f7cdbce" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
I20260928 02:46:10.580158 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58cdd8fe631466bbea770499f7cdbce" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843cabb00a84a889da46ace27bc5717"
I20260928 02:46:10.580256 31571 raft_consensus.cc:3037] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.580259 31711 raft_consensus.cc:3037] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.581176 31711 raft_consensus.cc:2454] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 1.
I20260928 02:46:10.581272 31571 raft_consensus.cc:2454] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 1.
I20260928 02:46:10.581470 31783 leader_election.cc:302] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b781616a99441aae68f5849d400daf, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:10.581621 31927 raft_consensus.cc:2781] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:10.581710 31927 raft_consensus.cc:686] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [term 1 LEADER]: Becoming Leader. State: Replica: a2b781616a99441aae68f5849d400daf, State: Running, Role: LEADER
I20260928 02:46:10.581853 31927 consensus_queue.cc:237] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.582490 31930 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: TxnStatusTablet state changed. Reason: New leader a2b781616a99441aae68f5849d400daf. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.582599 31930 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:10.582914 31942 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:46:10.583130 31942 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:46:10.583379 31412 catalog_manager.cc:5495] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b781616a99441aae68f5849d400daf (127.28.248.131). New cstate: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } health_report { overall_health: HEALTHY } } }
I20260928 02:46:10.595741 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.591884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41639 (local address 127.0.0.1:55396)
0928 02:46:10.592055 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.592075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.592180 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.592516 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.592519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.592526 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:10.592670 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.592677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.593466 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.593470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.594104 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.594112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.594645 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.594649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.594762 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.594803 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:10.595435 (+ 632us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:10.595450 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.595517 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.595615 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:46:10.595753 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.591997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.28.248.131:41639)
0928 02:46:10.592143 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.592148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.592161 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:10.592352 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.592359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.592408 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.592503 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:10.592791 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.593330 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.594240 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.594533 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.594648 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.594943 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:10.595342 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:10.595460 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.595525 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.595576 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:46:10.601238 31711 raft_consensus.cc:1261] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Refusing update from remote peer a2b781616a99441aae68f5849d400daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.601238 31571 raft_consensus.cc:1261] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Refusing update from remote peer a2b781616a99441aae68f5849d400daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 02:46:10.601986 31758 tablet.cc:2369] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:10.601958 31927 consensus_queue.cc:1035] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:10.602133 31930 consensus_queue.cc:1035] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:10.606070 31926 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: TxnStatusTablet state changed. Reason: New leader a2b781616a99441aae68f5849d400daf. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.606190 31926 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.605705 31928 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: TxnStatusTablet state changed. Reason: New leader a2b781616a99441aae68f5849d400daf. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.606371 31928 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.610309 31930 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.610447 31930 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:10.611383 31926 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.611487 31926 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P c843cabb00a84a889da46ace27bc5717: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.613996 31945 mvcc.cc:205] Tried to move back new op lower bound from 7334148385175404544 to 7334148385104560128. Current Snapshot: MvccSnapshot[applied={T|T < 7334148385175404544 or (T in {7334148385175404544})}]
I20260928 02:46:10.619067 31927 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.619185 31927 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P a2b781616a99441aae68f5849d400daf: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:10.621001 31947 mvcc.cc:205] Tried to move back new op lower bound from 7334148385175404544 to 7334148385104560128. Current Snapshot: MvccSnapshot[applied={T|T < 7334148385175404544}]
I20260928 02:46:10.621011 31928 tablet_replica.cc:439] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } } }
I20260928 02:46:10.621187 31928 tablet_replica.cc:442] T b58cdd8fe631466bbea770499f7cdbce P 2f8d03b7b12e48dab2b1507ed71aec8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:10.621359 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.617037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.28.248.131:41639)
0928 02:46:10.617145 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.617149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.617266 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:46:10.617446 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.617449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.617495 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.617578 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:10.617850 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.618359 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.619525 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.619865 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.619966 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.620571 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:10.620687 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:10.620767 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.621178 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.621233 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:46:10.631867 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.628313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41639 (local address 127.28.248.131:40381)
0928 02:46:10.628523 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.628537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.628679 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.629126 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.629131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.629141 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:10.629282 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.629290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.630011 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.630014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.630659 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.630667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.631087 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.631090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.631202 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.631255 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:10.631558 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:10.631566 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.631629 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.631702 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":22}
I20260928 02:46:10.631867 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.628350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:40381 (local address 127.28.248.131:41639)
0928 02:46:10.628453 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.628459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.628654 (+ 195us) server_negotiation.cc:407] Connection header received
0928 02:46:10.628760 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.628763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.628809 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.628885 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:10.629376 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.629899 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.630774 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.630986 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.631090 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.631403 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:10.631477 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:10.631557 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.631635 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.631678 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33}
I20260928 02:46:10.639509 31711 raft_consensus.cc:1261] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Refusing update from remote peer a2b781616a99441aae68f5849d400daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.639493 31571 raft_consensus.cc:1261] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Refusing update from remote peer a2b781616a99441aae68f5849d400daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.639988 31927 consensus_queue.cc:1035] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:10.640158 31930 consensus_queue.cc:1035] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:46:10.640851 31618 tablet.cc:2369] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:10.647432 31946 mvcc.cc:205] Tried to move back new op lower bound from 7334148385335873536 to 7334148384799199232. Current Snapshot: MvccSnapshot[applied={T|T < 7334148385335873536}]
I20260928 02:46:10.684911 31412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:10.685110 31412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:10.687726 31412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Sending DeleteTablet for 3 replicas of tablet 226c406c822e40aba4462891ebe75040
I20260928 02:46:10.688444 31831 tablet_service.cc:1514] Processing DeleteTablet for tablet 226c406c822e40aba4462891ebe75040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:10 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55380
I20260928 02:46:10.688778 31691 tablet_service.cc:1514] Processing DeleteTablet for tablet 226c406c822e40aba4462891ebe75040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:10 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:57892
I20260928 02:46:10.689160 31961 tablet_replica.cc:330] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: stopping tablet replica
I20260928 02:46:10.688555 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet 226c406c822e40aba4462891ebe75040 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:10 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:52892
I20260928 02:46:10.689539 31963 tablet_replica.cc:330] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: stopping tablet replica
I20260928 02:46:10.689646 31961 raft_consensus.cc:2229] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:10.689867 31963 raft_consensus.cc:2229] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:10.689816 31412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:46:10.690091 31963 raft_consensus.cc:2258] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:46:10.690238 31412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:10.690655 31961 raft_consensus.cc:2258] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:10.692047 31963 ts_tablet_manager.cc:1905] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:10.692274 31961 ts_tablet_manager.cc:1905] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:10.695276 31963 ts_tablet_manager.cc:1918] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:10.695370 31963 log.cc:1196] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/226c406c822e40aba4462891ebe75040
I20260928 02:46:10.695405 31961 ts_tablet_manager.cc:1918] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:10.695495 31961 log.cc:1196] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/226c406c822e40aba4462891ebe75040
I20260928 02:46:10.695859 31961 ts_tablet_manager.cc:1939] T 226c406c822e40aba4462891ebe75040 P a2b781616a99441aae68f5849d400daf: Deleting consensus metadata
I20260928 02:46:10.695813 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9737fac5de41b3a132f366a1c68cf0 (DEFAULT_TABLE table=test-table [id=f66d69b71feb42d0aab9104edf99266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.696048 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9737fac5de41b3a132f366a1c68cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.696545 31831 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9737fac5de41b3a132f366a1c68cf0 (DEFAULT_TABLE table=test-table [id=f66d69b71feb42d0aab9104edf99266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.696817 31831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9737fac5de41b3a132f366a1c68cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.696861 31691 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9737fac5de41b3a132f366a1c68cf0 (DEFAULT_TABLE table=test-table [id=f66d69b71feb42d0aab9104edf99266c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.697126 31691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9737fac5de41b3a132f366a1c68cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.698714 31921 tablet_bootstrap.cc:492] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Bootstrap starting.
I20260928 02:46:10.699433 31920 tablet_bootstrap.cc:492] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Bootstrap starting.
I20260928 02:46:10.699590 31921 tablet_bootstrap.cc:654] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.700389 31920 tablet_bootstrap.cc:654] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.700696 31921 tablet_bootstrap.cc:492] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: No bootstrap required, opened a new log
I20260928 02:46:10.700778 31921 ts_tablet_manager.cc:1397] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:10.700785 31962 tablet_replica.cc:330] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: stopping tablet replica
I20260928 02:46:10.700881 31398 catalog_manager.cc:4841] TS a2b781616a99441aae68f5849d400daf (127.28.248.131:41639): tablet 226c406c822e40aba4462891ebe75040 (table test-table [id=c28f3d3356ee46a2b68157dc8cc192bc]) successfully deleted
I20260928 02:46:10.701128 31962 raft_consensus.cc:2229] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:10.701227 31921 raft_consensus.cc:348] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.701344 31921 raft_consensus.cc:374] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.701346 31962 raft_consensus.cc:2258] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:10.701385 31921 raft_consensus.cc:729] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b781616a99441aae68f5849d400daf, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.701505 31921 consensus_queue.cc:260] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.701790 31921 ts_tablet_manager.cc:1428] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:10.702961 31922 tablet_bootstrap.cc:492] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Bootstrap starting.
I20260928 02:46:10.703611 31963 ts_tablet_manager.cc:1939] T 226c406c822e40aba4462891ebe75040 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting consensus metadata
I20260928 02:46:10.703972 31922 tablet_bootstrap.cc:654] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.704347 31962 ts_tablet_manager.cc:1905] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:10.704859 31399 catalog_manager.cc:4841] TS 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677): tablet 226c406c822e40aba4462891ebe75040 (table test-table [id=c28f3d3356ee46a2b68157dc8cc192bc]) successfully deleted
I20260928 02:46:10.705181 31922 tablet_bootstrap.cc:492] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: No bootstrap required, opened a new log
I20260928 02:46:10.705252 31922 ts_tablet_manager.cc:1397] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:10.705693 31922 raft_consensus.cc:348] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.705840 31922 raft_consensus.cc:374] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.705883 31922 raft_consensus.cc:729] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d03b7b12e48dab2b1507ed71aec8e, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.706007 31922 consensus_queue.cc:260] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.706336 31922 ts_tablet_manager.cc:1428] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:10.706629 31920 tablet_bootstrap.cc:492] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: No bootstrap required, opened a new log
I20260928 02:46:10.706688 31920 ts_tablet_manager.cc:1397] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 02:46:10.707180 31920 raft_consensus.cc:348] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.707261 31920 raft_consensus.cc:374] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.707283 31920 raft_consensus.cc:729] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.707324 31962 ts_tablet_manager.cc:1918] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:10.707435 31962 log.cc:1196] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/226c406c822e40aba4462891ebe75040
I20260928 02:46:10.707415 31920 consensus_queue.cc:260] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.707692 31920 ts_tablet_manager.cc:1428] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:10.707751 31962 ts_tablet_manager.cc:1939] T 226c406c822e40aba4462891ebe75040 P c843cabb00a84a889da46ace27bc5717: Deleting consensus metadata
I20260928 02:46:10.708988 31399 catalog_manager.cc:4841] TS c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177): tablet 226c406c822e40aba4462891ebe75040 (table test-table [id=c28f3d3356ee46a2b68157dc8cc192bc]) successfully deleted
I20260928 02:46:10.818323 31960 raft_consensus.cc:482] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:10.818500 31960 raft_consensus.cc:504] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.818957 31960 leader_election.cc:288] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177)
I20260928 02:46:10.819434 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9737fac5de41b3a132f366a1c68cf0" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" is_pre_election: true
I20260928 02:46:10.819482 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9737fac5de41b3a132f366a1c68cf0" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843cabb00a84a889da46ace27bc5717" is_pre_election: true
I20260928 02:46:10.819600 31571 raft_consensus.cc:2454] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 0.
I20260928 02:46:10.819612 31711 raft_consensus.cc:2454] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 0.
I20260928 02:46:10.819934 31783 leader_election.cc:302] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d03b7b12e48dab2b1507ed71aec8e, a2b781616a99441aae68f5849d400daf; no voters:
I20260928 02:46:10.820127 31960 raft_consensus.cc:2781] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:10.820185 31960 raft_consensus.cc:482] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:10.820225 31960 raft_consensus.cc:3037] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.821285 31960 raft_consensus.cc:504] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.821602 31960 leader_election.cc:288] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 election: Requested vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177)
I20260928 02:46:10.821920 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9737fac5de41b3a132f366a1c68cf0" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
I20260928 02:46:10.822052 31571 raft_consensus.cc:3037] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.822006 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9737fac5de41b3a132f366a1c68cf0" candidate_uuid: "a2b781616a99441aae68f5849d400daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c843cabb00a84a889da46ace27bc5717"
I20260928 02:46:10.822152 31711 raft_consensus.cc:3037] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.823129 31711 raft_consensus.cc:2454] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 1.
I20260928 02:46:10.823158 31571 raft_consensus.cc:2454] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b781616a99441aae68f5849d400daf in term 1.
I20260928 02:46:10.823490 31783 leader_election.cc:302] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d03b7b12e48dab2b1507ed71aec8e, a2b781616a99441aae68f5849d400daf; no voters:
I20260928 02:46:10.823652 31960 raft_consensus.cc:2781] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:10.823747 31960 raft_consensus.cc:686] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 1 LEADER]: Becoming Leader. State: Replica: a2b781616a99441aae68f5849d400daf, State: Running, Role: LEADER
I20260928 02:46:10.823882 31960 consensus_queue.cc:237] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:10.825695 31413 catalog_manager.cc:5495] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b781616a99441aae68f5849d400daf (127.28.248.131). New cstate: current_term: 1 leader_uuid: "a2b781616a99441aae68f5849d400daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } health_report { overall_health: HEALTHY } } }
I20260928 02:46:10.872421 31711 raft_consensus.cc:1261] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Refusing update from remote peer a2b781616a99441aae68f5849d400daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.872428 31571 raft_consensus.cc:1261] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Refusing update from remote peer a2b781616a99441aae68f5849d400daf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.872821 31927 consensus_queue.cc:1035] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:10.872977 31960 consensus_queue.cc:1035] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:46:10.886513 31945 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:46:12.444268 31413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:12.444448 31413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:12.446864 31413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Sending DeleteTablet for 3 replicas of tablet 0c9737fac5de41b3a132f366a1c68cf0
I20260928 02:46:12.447463 31691 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9737fac5de41b3a132f366a1c68cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:12 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:57892
I20260928 02:46:12.447477 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9737fac5de41b3a132f366a1c68cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:12 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:52892
I20260928 02:46:12.447516 31824 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9737fac5de41b3a132f366a1c68cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:12 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55380
I20260928 02:46:12.447851 31992 tablet_replica.cc:330] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: stopping tablet replica
I20260928 02:46:12.448045 31992 raft_consensus.cc:2229] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:12.448184 31992 raft_consensus.cc:2258] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:12.448439 31993 tablet_replica.cc:330] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: stopping tablet replica
I20260928 02:46:12.448598 31993 raft_consensus.cc:2229] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:12.448745 31993 raft_consensus.cc:2258] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:12.449338 31992 ts_tablet_manager.cc:1905] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:12.451467 31992 ts_tablet_manager.cc:1918] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:46:12.451529 31992 log.cc:1196] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/0c9737fac5de41b3a132f366a1c68cf0
I20260928 02:46:12.451715 31992 ts_tablet_manager.cc:1939] T 0c9737fac5de41b3a132f366a1c68cf0 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting consensus metadata
I20260928 02:46:12.451915 31994 tablet_replica.cc:330] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: stopping tablet replica
I20260928 02:46:12.452078 31994 raft_consensus.cc:2229] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:12.452318 31994 raft_consensus.cc:2258] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:12.452554 31399 catalog_manager.cc:4841] TS 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677): tablet 0c9737fac5de41b3a132f366a1c68cf0 (table test-table [id=f66d69b71feb42d0aab9104edf99266c]) successfully deleted
I20260928 02:46:12.453028 31993 ts_tablet_manager.cc:1905] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:12.453078 31994 ts_tablet_manager.cc:1905] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:12.455466 31993 ts_tablet_manager.cc:1918] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:46:12.455466 31994 ts_tablet_manager.cc:1918] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:46:12.455545 31994 log.cc:1196] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/0c9737fac5de41b3a132f366a1c68cf0
I20260928 02:46:12.455545 31993 log.cc:1196] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/0c9737fac5de41b3a132f366a1c68cf0
I20260928 02:46:12.455807 31993 ts_tablet_manager.cc:1939] T 0c9737fac5de41b3a132f366a1c68cf0 P c843cabb00a84a889da46ace27bc5717: Deleting consensus metadata
I20260928 02:46:12.455825 31994 ts_tablet_manager.cc:1939] T 0c9737fac5de41b3a132f366a1c68cf0 P a2b781616a99441aae68f5849d400daf: Deleting consensus metadata
I20260928 02:46:12.456511 31398 catalog_manager.cc:4841] TS a2b781616a99441aae68f5849d400daf (127.28.248.131:41639): tablet 0c9737fac5de41b3a132f366a1c68cf0 (table test-table [id=f66d69b71feb42d0aab9104edf99266c]) successfully deleted
I20260928 02:46:12.456624 31399 catalog_manager.cc:4841] TS c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177): tablet 0c9737fac5de41b3a132f366a1c68cf0 (table test-table [id=f66d69b71feb42d0aab9104edf99266c]) successfully deleted
Sep 28 02:46:12 dist-test-slave-r238 krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:46:12 dist-test-slave-r238 krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:12.481451 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.466383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.28.248.190:39719)
0928 02:46:12.466669 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.466681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.466708 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:46:12.466899 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.466903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.466953 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.467052 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.467757 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.468311 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.469263 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.469508 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.469620 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.478121 (+ 8501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.478125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.478127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.478157 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.479568 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.480015 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.480018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.480020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.480106 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.480384 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.480388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.480390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.480710 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.480857 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.481110 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.481266 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260928 02:46:12.484083 31413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:12.484534 31413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:12.489472 31691 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5b1841abf14cf5afad30b613234546 (DEFAULT_TABLE table=test-table [id=4a9839dd139b4989ba781a3dc2bea50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:12.489472 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5b1841abf14cf5afad30b613234546 (DEFAULT_TABLE table=test-table [id=4a9839dd139b4989ba781a3dc2bea50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:12.489625 31824 tablet_service.cc:1467] Processing CreateTablet for tablet 0f5b1841abf14cf5afad30b613234546 (DEFAULT_TABLE table=test-table [id=4a9839dd139b4989ba781a3dc2bea50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:12.489773 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5b1841abf14cf5afad30b613234546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.489773 31691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5b1841abf14cf5afad30b613234546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.489964 31824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f5b1841abf14cf5afad30b613234546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.492316 32004 tablet_bootstrap.cc:492] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Bootstrap starting.
I20260928 02:46:12.492630 32005 tablet_bootstrap.cc:492] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Bootstrap starting.
I20260928 02:46:12.493593 32004 tablet_bootstrap.cc:654] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.493593 32005 tablet_bootstrap.cc:654] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.494844 32003 tablet_bootstrap.cc:492] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Bootstrap starting.
I20260928 02:46:12.494988 32005 tablet_bootstrap.cc:492] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: No bootstrap required, opened a new log
I20260928 02:46:12.495083 32005 ts_tablet_manager.cc:1397] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:12.495507 32005 raft_consensus.cc:348] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.495616 32005 raft_consensus.cc:374] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.495638 32005 raft_consensus.cc:729] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b781616a99441aae68f5849d400daf, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.495764 32005 consensus_queue.cc:260] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.496057 32005 ts_tablet_manager.cc:1428] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:12.496366 32003 tablet_bootstrap.cc:654] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.497063 32004 tablet_bootstrap.cc:492] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: No bootstrap required, opened a new log
I20260928 02:46:12.497162 32004 ts_tablet_manager.cc:1397] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 02:46:12.497668 32004 raft_consensus.cc:348] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.497814 32004 raft_consensus.cc:374] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.497856 32004 raft_consensus.cc:729] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d03b7b12e48dab2b1507ed71aec8e, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.497875 32003 tablet_bootstrap.cc:492] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: No bootstrap required, opened a new log
I20260928 02:46:12.497965 32003 ts_tablet_manager.cc:1397] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:12.498003 32004 consensus_queue.cc:260] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.498427 32004 ts_tablet_manager.cc:1428] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:12.498471 32003 raft_consensus.cc:348] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.498607 32003 raft_consensus.cc:374] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.498656 32003 raft_consensus.cc:729] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.498824 32003 consensus_queue.cc:260] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.499385 32003 ts_tablet_manager.cc:1428] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:12.622368 32010 raft_consensus.cc:482] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:12.622527 32010 raft_consensus.cc:504] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.623771 32010 leader_election.cc:288] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), a2b781616a99441aae68f5849d400daf (127.28.248.131:41639)
I20260928 02:46:12.628064 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.624244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:52597 (local address 127.28.248.131:41639)
0928 02:46:12.624555 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.624571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.624593 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:46:12.624752 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.624757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.624818 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.624899 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:12.625214 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.625755 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.626995 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.627378 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.627507 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.627563 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.627852 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.627933 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":130,"threads_started":1}
I20260928 02:46:12.628073 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.624142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:40705 (local address 127.28.248.129:45677)
0928 02:46:12.624528 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.624538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.624666 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:46:12.624818 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.624820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.624867 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.624977 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:12.625267 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.625789 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.626962 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.627385 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.627526 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.627578 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.627857 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.627932 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":105,"threads_started":1}
I20260928 02:46:12.628077 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.624014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41639 (local address 127.28.248.130:52597)
0928 02:46:12.624444 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.624463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.624582 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.624934 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.624940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.624962 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:12.625104 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.625110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.625900 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.625904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.626823 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.626832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.627519 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.627523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.627684 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.627717 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.627776 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.627879 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":113,"thread_start_us":141,"threads_started":1}
I20260928 02:46:12.628080 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.624014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:45677 (local address 127.28.248.130:40705)
0928 02:46:12.624554 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.624570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.624714 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.625011 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.625014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.625031 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:12.625136 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.625141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.625956 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.625959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.626819 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.626828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.627529 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.627533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.627714 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.627731 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.627828 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.627906 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":82,"thread_start_us":113,"threads_started":1}
I20260928 02:46:12.628737 31851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5b1841abf14cf5afad30b613234546" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b781616a99441aae68f5849d400daf" is_pre_election: true
I20260928 02:46:12.629029 31851 raft_consensus.cc:2454] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 0.
I20260928 02:46:12.629143 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5b1841abf14cf5afad30b613234546" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" is_pre_election: true
I20260928 02:46:12.629277 31571 raft_consensus.cc:2454] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 0.
I20260928 02:46:12.629452 31642 leader_election.cc:302] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b781616a99441aae68f5849d400daf, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:12.629767 32010 raft_consensus.cc:2781] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:12.629820 32010 raft_consensus.cc:482] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:12.629873 32010 raft_consensus.cc:3037] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.630857 32010 raft_consensus.cc:504] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.631233 32010 leader_election.cc:288] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 election: Requested vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), a2b781616a99441aae68f5849d400daf (127.28.248.131:41639)
I20260928 02:46:12.631515 31571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5b1841abf14cf5afad30b613234546" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
I20260928 02:46:12.631632 31571 raft_consensus.cc:3037] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.631634 31851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f5b1841abf14cf5afad30b613234546" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b781616a99441aae68f5849d400daf"
I20260928 02:46:12.631752 31851 raft_consensus.cc:3037] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.632562 31571 raft_consensus.cc:2454] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 1.
I20260928 02:46:12.632788 31851 raft_consensus.cc:2454] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 1.
I20260928 02:46:12.632865 31643 leader_election.cc:302] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d03b7b12e48dab2b1507ed71aec8e, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:12.633061 32010 raft_consensus.cc:2781] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:12.633311 32010 raft_consensus.cc:686] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 1 LEADER]: Becoming Leader. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Running, Role: LEADER
I20260928 02:46:12.633752 32010 consensus_queue.cc:237] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.636229 31413 catalog_manager.cc:5495] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 reported cstate change: term changed from 0 to 1, leader changed from <none> to c843cabb00a84a889da46ace27bc5717 (127.28.248.130). New cstate: current_term: 1 leader_uuid: "c843cabb00a84a889da46ace27bc5717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:12.656306 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.652842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.28.248.130:33177)
0928 02:46:12.653082 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.653092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.653103 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:12.653168 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.653171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.653219 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.653315 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:12.653521 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.654070 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.654857 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.655090 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.655163 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.655377 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:12.655826 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:12.655936 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.656148 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.656193 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
I20260928 02:46:12.666563 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.662931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41639 (local address 127.28.248.130:39137)
0928 02:46:12.663296 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.663308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.663426 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.663676 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.663678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.663685 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:12.663812 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.663818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.664548 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.664554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.665315 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.665324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.665813 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.665817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.665913 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.665954 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:12.666272 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:12.666278 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.666334 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.666411 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260928 02:46:12.666551 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.663061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:39137 (local address 127.28.248.131:41639)
0928 02:46:12.663205 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.663210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.663378 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:46:12.663534 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.663541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.663585 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.663668 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:12.663898 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.664449 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.665466 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.665689 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.665805 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.666107 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:12.666178 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:12.666256 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.666354 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.666409 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:12.672851 31571 raft_consensus.cc:1261] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Refusing update from remote peer c843cabb00a84a889da46ace27bc5717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:12.672858 31851 raft_consensus.cc:1261] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Refusing update from remote peer c843cabb00a84a889da46ace27bc5717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:12.673370 32015 consensus_queue.cc:1035] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:12.673554 32010 consensus_queue.cc:1035] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:12.690855 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.687303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54987 (local address 127.28.248.130:33177)
0928 02:46:12.687737 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.687741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.687754 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:12.687801 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.687803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.687848 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.687944 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:12.688201 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.688784 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.689632 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.689937 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.690028 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.690343 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:12.690447 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:12.690520 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.690628 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.690699 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260928 02:46:12.690879 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.687247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33177 (local address 127.28.248.131:54987)
0928 02:46:12.687539 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.687552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.687629 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.687958 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.687961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.687973 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:12.688096 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.688103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.688879 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.688882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.689508 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.689516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.690055 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.690059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.690150 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.690190 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:12.690538 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:12.690545 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.690611 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.690700 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
I20260928 02:46:12.713933 31413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:12.714087 31413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:12.716288 31413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Sending DeleteTablet for 3 replicas of tablet 0f5b1841abf14cf5afad30b613234546
I20260928 02:46:12.716957 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5b1841abf14cf5afad30b613234546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:12 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:52892
I20260928 02:46:12.717010 31824 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5b1841abf14cf5afad30b613234546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:12 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55380
I20260928 02:46:12.717207 31992 tablet_replica.cc:330] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: stopping tablet replica
I20260928 02:46:12.717272 31691 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f5b1841abf14cf5afad30b613234546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:12 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:57892
I20260928 02:46:12.717332 31992 raft_consensus.cc:2229] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:12.717438 31993 tablet_replica.cc:330] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: stopping tablet replica
I20260928 02:46:12.717489 31992 raft_consensus.cc:2258] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:12.717484 31994 tablet_replica.cc:330] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: stopping tablet replica
I20260928 02:46:12.717543 31993 raft_consensus.cc:2229] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:12.717623 31994 raft_consensus.cc:2229] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:12.717741 31994 raft_consensus.cc:2258] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:12.717792 31993 raft_consensus.cc:2258] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:12.718493 31994 ts_tablet_manager.cc:1905] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:12.718549 31993 ts_tablet_manager.cc:1905] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:12.719034 31413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:12.719441 31413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:12.721233 31994 ts_tablet_manager.cc:1918] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:12.721303 31994 log.cc:1196] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/0f5b1841abf14cf5afad30b613234546
I20260928 02:46:12.721508 31992 ts_tablet_manager.cc:1905] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:12.721572 31994 ts_tablet_manager.cc:1939] T 0f5b1841abf14cf5afad30b613234546 P a2b781616a99441aae68f5849d400daf: Deleting consensus metadata
I20260928 02:46:12.721879 31993 ts_tablet_manager.cc:1918] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:12.721944 31993 log.cc:1196] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/0f5b1841abf14cf5afad30b613234546
I20260928 02:46:12.722173 31993 ts_tablet_manager.cc:1939] T 0f5b1841abf14cf5afad30b613234546 P c843cabb00a84a889da46ace27bc5717: Deleting consensus metadata
I20260928 02:46:12.722254 31398 catalog_manager.cc:4841] TS a2b781616a99441aae68f5849d400daf (127.28.248.131:41639): tablet 0f5b1841abf14cf5afad30b613234546 (table test-table [id=4a9839dd139b4989ba781a3dc2bea50b]) successfully deleted
I20260928 02:46:12.722911 31399 catalog_manager.cc:4841] TS c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177): tablet 0f5b1841abf14cf5afad30b613234546 (table test-table [id=4a9839dd139b4989ba781a3dc2bea50b]) successfully deleted
I20260928 02:46:12.724076 31992 ts_tablet_manager.cc:1918] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:12.724154 31992 log.cc:1196] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/0f5b1841abf14cf5afad30b613234546
I20260928 02:46:12.724416 31992 ts_tablet_manager.cc:1939] T 0f5b1841abf14cf5afad30b613234546 P 2f8d03b7b12e48dab2b1507ed71aec8e: Deleting consensus metadata
I20260928 02:46:12.725071 31399 catalog_manager.cc:4841] TS 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677): tablet 0f5b1841abf14cf5afad30b613234546 (table test-table [id=4a9839dd139b4989ba781a3dc2bea50b]) successfully deleted
I20260928 02:46:12.725760 31551 tablet_service.cc:1467] Processing CreateTablet for tablet 2a49399362a14c5eb7fce7d696345511 (DEFAULT_TABLE table=test-table [id=469fed4b6a9942d2b2a3f4b62d0a2780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:12.725761 31824 tablet_service.cc:1467] Processing CreateTablet for tablet 2a49399362a14c5eb7fce7d696345511 (DEFAULT_TABLE table=test-table [id=469fed4b6a9942d2b2a3f4b62d0a2780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:12.725878 31691 tablet_service.cc:1467] Processing CreateTablet for tablet 2a49399362a14c5eb7fce7d696345511 (DEFAULT_TABLE table=test-table [id=469fed4b6a9942d2b2a3f4b62d0a2780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:12.726006 31551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a49399362a14c5eb7fce7d696345511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.726019 31824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a49399362a14c5eb7fce7d696345511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.726089 31691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a49399362a14c5eb7fce7d696345511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.728190 32004 tablet_bootstrap.cc:492] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e: Bootstrap starting.
I20260928 02:46:12.728210 32003 tablet_bootstrap.cc:492] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Bootstrap starting.
I20260928 02:46:12.729128 32004 tablet_bootstrap.cc:654] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.729203 32003 tablet_bootstrap.cc:654] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.730042 32005 tablet_bootstrap.cc:492] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Bootstrap starting.
I20260928 02:46:12.730792 32004 tablet_bootstrap.cc:492] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e: No bootstrap required, opened a new log
I20260928 02:46:12.730867 32004 ts_tablet_manager.cc:1397] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:12.731021 32005 tablet_bootstrap.cc:654] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.731334 32004 raft_consensus.cc:348] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.731468 32004 raft_consensus.cc:374] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.731501 32004 raft_consensus.cc:729] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8d03b7b12e48dab2b1507ed71aec8e, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.731621 32004 consensus_queue.cc:260] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.731886 32004 ts_tablet_manager.cc:1428] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:12.732071 32003 tablet_bootstrap.cc:492] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: No bootstrap required, opened a new log
I20260928 02:46:12.732144 32003 ts_tablet_manager.cc:1397] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 02:46:12.732596 32003 raft_consensus.cc:348] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.732728 32003 raft_consensus.cc:374] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.732762 32003 raft_consensus.cc:729] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.732918 32003 consensus_queue.cc:260] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.734093 32005 tablet_bootstrap.cc:492] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: No bootstrap required, opened a new log
I20260928 02:46:12.734175 32005 ts_tablet_manager.cc:1397] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:46:12.734577 32005 raft_consensus.cc:348] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.734675 32005 raft_consensus.cc:374] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.734699 32005 raft_consensus.cc:729] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b781616a99441aae68f5849d400daf, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.734791 32005 consensus_queue.cc:260] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.735039 32005 ts_tablet_manager.cc:1428] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:12.735400 32003 ts_tablet_manager.cc:1428] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:46:12.783310 32026 raft_consensus.cc:482] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:12.783480 32026 raft_consensus.cc:504] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.783929 32026 leader_election.cc:288] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), a2b781616a99441aae68f5849d400daf (127.28.248.131:41639)
I20260928 02:46:12.784406 31570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49399362a14c5eb7fce7d696345511" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" is_pre_election: true
I20260928 02:46:12.784426 31851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49399362a14c5eb7fce7d696345511" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b781616a99441aae68f5849d400daf" is_pre_election: true
I20260928 02:46:12.784590 31851 raft_consensus.cc:2454] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 0.
I20260928 02:46:12.784605 31570 raft_consensus.cc:2454] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 0.
I20260928 02:46:12.784953 31642 leader_election.cc:302] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b781616a99441aae68f5849d400daf, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:12.785228 32026 raft_consensus.cc:2781] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:12.785292 32026 raft_consensus.cc:482] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:12.785348 32026 raft_consensus.cc:3037] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.786901 32026 raft_consensus.cc:504] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.787580 32026 leader_election.cc:288] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 election: Requested vote from peers 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677), a2b781616a99441aae68f5849d400daf (127.28.248.131:41639)
I20260928 02:46:12.787768 31570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49399362a14c5eb7fce7d696345511" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e"
I20260928 02:46:12.787900 31570 raft_consensus.cc:3037] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.788174 31851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49399362a14c5eb7fce7d696345511" candidate_uuid: "c843cabb00a84a889da46ace27bc5717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b781616a99441aae68f5849d400daf"
I20260928 02:46:12.788355 31851 raft_consensus.cc:3037] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.789251 31570 raft_consensus.cc:2454] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 1.
I20260928 02:46:12.789562 31643 leader_election.cc:302] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8d03b7b12e48dab2b1507ed71aec8e, c843cabb00a84a889da46ace27bc5717; no voters:
I20260928 02:46:12.789762 32026 raft_consensus.cc:2781] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:12.789951 32026 raft_consensus.cc:686] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 1 LEADER]: Becoming Leader. State: Replica: c843cabb00a84a889da46ace27bc5717, State: Running, Role: LEADER
I20260928 02:46:12.789937 31851 raft_consensus.cc:2454] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c843cabb00a84a889da46ace27bc5717 in term 1.
I20260928 02:46:12.790089 32026 consensus_queue.cc:237] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } }
I20260928 02:46:12.792018 31412 catalog_manager.cc:5495] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 reported cstate change: term changed from 0 to 1, leader changed from <none> to c843cabb00a84a889da46ace27bc5717 (127.28.248.130). New cstate: current_term: 1 leader_uuid: "c843cabb00a84a889da46ace27bc5717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c843cabb00a84a889da46ace27bc5717" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:12.823822 31851 raft_consensus.cc:1261] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Refusing update from remote peer c843cabb00a84a889da46ace27bc5717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:12.823832 31570 raft_consensus.cc:1261] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Refusing update from remote peer c843cabb00a84a889da46ace27bc5717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:12.824239 32026 consensus_queue.cc:1035] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b781616a99441aae68f5849d400daf" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:12.824486 32015 consensus_queue.cc:1035] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8d03b7b12e48dab2b1507ed71aec8e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 45677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:46:12.839524 31976 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:46:14.392122 31412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:14.392292 31412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:14.395109 31412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb74a9318934ee6b7fd311093df5c3c: Sending DeleteTablet for 3 replicas of tablet 2a49399362a14c5eb7fce7d696345511
I20260928 02:46:14.395622 31551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a49399362a14c5eb7fce7d696345511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:14 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:52892
I20260928 02:46:14.395658 31691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a49399362a14c5eb7fce7d696345511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:14 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:57892
I20260928 02:46:14.395740 31824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a49399362a14c5eb7fce7d696345511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:14 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55380
I20260928 02:46:14.396098 32058 tablet_replica.cc:330] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: stopping tablet replica
I20260928 02:46:14.396098 32056 tablet_replica.cc:330] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e: stopping tablet replica
I20260928 02:46:14.396226 32056 raft_consensus.cc:2229] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:14.396284 32058 raft_consensus.cc:2229] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:14.396347 32056 raft_consensus.cc:2258] T 2a49399362a14c5eb7fce7d696345511 P 2f8d03b7b12e48dab2b1507ed71aec8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:14.396385 32058 raft_consensus.cc:2258] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:14.396442 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 31486
I20260928 02:46:14.398137 32057 tablet_replica.cc:330] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: stopping tablet replica
I20260928 02:46:14.398307 32057 raft_consensus.cc:2229] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:14.398524 32058 ts_tablet_manager.cc:1905] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:14.398521 32057 raft_consensus.cc:2258] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:14.399185 32057 ts_tablet_manager.cc:1905] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:14.400825 32058 ts_tablet_manager.cc:1918] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:46:14.400921 32058 log.cc:1196] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/2a49399362a14c5eb7fce7d696345511
I20260928 02:46:14.401168 32058 ts_tablet_manager.cc:1939] T 2a49399362a14c5eb7fce7d696345511 P a2b781616a99441aae68f5849d400daf: Deleting consensus metadata
I20260928 02:46:14.401839 31398 catalog_manager.cc:4841] TS a2b781616a99441aae68f5849d400daf (127.28.248.131:41639): tablet 2a49399362a14c5eb7fce7d696345511 (table test-table [id=469fed4b6a9942d2b2a3f4b62d0a2780]) successfully deleted
I20260928 02:46:14.402562 32057 ts_tablet_manager.cc:1918] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:46:14.402647 32057 log.cc:1196] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/2a49399362a14c5eb7fce7d696345511
I20260928 02:46:14.402931 32057 ts_tablet_manager.cc:1939] T 2a49399362a14c5eb7fce7d696345511 P c843cabb00a84a889da46ace27bc5717: Deleting consensus metadata
I20260928 02:46:14.403656 31399 catalog_manager.cc:4841] TS c843cabb00a84a889da46ace27bc5717 (127.28.248.130:33177): tablet 2a49399362a14c5eb7fce7d696345511 (table test-table [id=469fed4b6a9942d2b2a3f4b62d0a2780]) successfully deleted
W20260928 02:46:14.403959 31399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:45677 (error 108)
I20260928 02:46:14.404291 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 31626
I20260928 02:46:14.406451 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:14.405992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:45677 (local address 127.0.0.1:39328)
0928 02:46:14.406239 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:14.406353 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
W20260928 02:46:14.406572 31399 catalog_manager.cc:4584] TS 2f8d03b7b12e48dab2b1507ed71aec8e (127.28.248.129:45677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a49399362a14c5eb7fce7d696345511: Network error: Client connection negotiation failed: client connection to 127.28.248.129:45677: connect: Connection refused (error 111)
I20260928 02:46:14.411751 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 31766
I20260928 02:46:14.418565 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 31382
2026-09-28T02:46:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6150 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:14 dist-test-slave-r238 krb5kdc[32063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:14 dist-test-slave-r238 krb5kdc[32063](info): set up 2 sockets
Sep 28 02:46:14 dist-test-slave-r238 krb5kdc[32063](info): commencing operation
krb5kdc: starting...
W20260928 02:46:15.500844 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:15 dist-test-slave-r238 krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:15.627975 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:42215
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:42215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:15.740291 32076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:15.740590 32076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:15.740724 32076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:15.744958 32076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:15.745061 32076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:15.745087 32076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:15.745110 32076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:15.745129 32076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:15.751147 32076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:42215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:42215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:15.752262 32076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:15.753340 32076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:15.763159 32082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:15.763226 32084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:15.763391 32076 server_base.cc:1034] running on GCE node
W20260928 02:46:15.763303 32081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:15.764484 32076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:15.765779 32076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:15.767139 32076 hybrid_clock.cc:648] HybridClock initialized: now 1790563575767122 us; error 37 us; skew 500 ppm
Sep 28 02:46:15 dist-test-slave-r238 krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563575, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:15.770335 32076 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:15.771373 32076 webserver.cc:460] Webserver started at http://127.28.248.190:45341/ using document root <none> and password file <none>
I20260928 02:46:15.772070 32076 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:15.772176 32076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:15.772431 32076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:15.774811 32076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "367adfd51e8a49948a5aa35cacc62dc7"
format_stamp: "Formatted at 2026-09-28 02:46:15 on dist-test-slave-r238"
server_key: "83d6495fd723c1e42f5b675d81bcf04d"
server_key_iv: "041ee8672a9d3353e1889eb7bf77c636"
server_key_version: "encryptionkey@0"
I20260928 02:46:15.775444 32076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "367adfd51e8a49948a5aa35cacc62dc7"
format_stamp: "Formatted at 2026-09-28 02:46:15 on dist-test-slave-r238"
server_key: "83d6495fd723c1e42f5b675d81bcf04d"
server_key_iv: "041ee8672a9d3353e1889eb7bf77c636"
server_key_version: "encryptionkey@0"
I20260928 02:46:15.779795 32076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:15.782773 32091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:15.784302 32076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:15.784468 32076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "367adfd51e8a49948a5aa35cacc62dc7"
format_stamp: "Formatted at 2026-09-28 02:46:15 on dist-test-slave-r238"
server_key: "83d6495fd723c1e42f5b675d81bcf04d"
server_key_iv: "041ee8672a9d3353e1889eb7bf77c636"
server_key_version: "encryptionkey@0"
I20260928 02:46:15.784600 32076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:15.803073 32076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:15.806366 32076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:15.806589 32076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:15.813779 32076 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:42215
I20260928 02:46:15.813796 32143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:42215 every 8 connection(s)
I20260928 02:46:15.814802 32076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:15.817723 32144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:15.823527 32144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7: Bootstrap starting.
I20260928 02:46:15.824740 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 32076
I20260928 02:46:15.824875 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:15.825156 29666 external_mini_cluster.cc:1428] Setting key a9fc6375fd09ebce05714d77ab96da67
I20260928 02:46:15.826483 32144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:15.827253 32144 log.cc:824] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:15.829521 32144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7: No bootstrap required, opened a new log
Sep 28 02:46:15 dist-test-slave-r238 krb5kdc[32063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:15.834052 32144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367adfd51e8a49948a5aa35cacc62dc7" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 42215 } }
I20260928 02:46:15.834312 32144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:15.834383 32144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367adfd51e8a49948a5aa35cacc62dc7, State: Initialized, Role: FOLLOWER
I20260928 02:46:15.834903 32144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367adfd51e8a49948a5aa35cacc62dc7" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 42215 } }
I20260928 02:46:15.835067 32144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:15.835145 32144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:15.835266 32144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:15.836581 32144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367adfd51e8a49948a5aa35cacc62dc7" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 42215 } }
I20260928 02:46:15.837005 32144 leader_election.cc:302] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 367adfd51e8a49948a5aa35cacc62dc7; no voters:
I20260928 02:46:15.837417 32144 leader_election.cc:288] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:15.837540 32149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:15.837788 32149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [term 1 LEADER]: Becoming Leader. State: Replica: 367adfd51e8a49948a5aa35cacc62dc7, State: Running, Role: LEADER
I20260928 02:46:15.838181 32149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367adfd51e8a49948a5aa35cacc62dc7" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 42215 } }
I20260928 02:46:15.838933 32144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:15.840488 32151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 367adfd51e8a49948a5aa35cacc62dc7. Latest consensus state: current_term: 1 leader_uuid: "367adfd51e8a49948a5aa35cacc62dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367adfd51e8a49948a5aa35cacc62dc7" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 42215 } } }
I20260928 02:46:15.840729 32151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:15.841363 32150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "367adfd51e8a49948a5aa35cacc62dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367adfd51e8a49948a5aa35cacc62dc7" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 42215 } } }
I20260928 02:46:15.841511 32150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:15.841637 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:15.826311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.28.248.190:42215)
0928 02:46:15.826730 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:15.826745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:15.826779 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:15.827484 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:15.827502 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:15.827937 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:15.828267 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:15.829158 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:15.830202 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:15.831393 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:15.831856 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:15.831997 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:15.834925 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:15.834932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:15.834949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:15.834983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:15.837200 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:15.837954 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:15.837961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:15.837969 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:15.838130 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:15.838503 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:15.838506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:15.838508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:15.839086 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:15.839299 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:15.839641 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:46:15.840032 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":140,"threads_started":1}
W20260928 02:46:15.844588 32164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:15.844743 32164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:15.844827 32163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:15.845551 32163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:15.851331 32163 catalog_manager.cc:1347] Generated new cluster ID: e4eadb56f9b743ba9021ec733fd91b38
I20260928 02:46:15.851424 32163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:15.860507 32163 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:15.861820 32163 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:15.867259 32163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 367adfd51e8a49948a5aa35cacc62dc7: Generated new TSK 0
I20260928 02:46:15.868141 32163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:15.899303 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:42215
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.013337 32170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.013695 32170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.013777 32170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.018020 32170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.018136 32170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.018235 32170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:16.024747 32170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:16.025907 32170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.026959 32170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.036585 32175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.036583 32176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.036648 32178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.037047 32170 server_base.cc:1034] running on GCE node
I20260928 02:46:16.037490 32170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.038189 32170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.039384 32170 hybrid_clock.cc:648] HybridClock initialized: now 1790563576039366 us; error 34 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.042438 32170 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:16.043311 32170 webserver.cc:460] Webserver started at http://127.28.248.129:36663/ using document root <none> and password file <none>
I20260928 02:46:16.043882 32170 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.043998 32170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.044238 32170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.046371 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "001d82435f004ef4813e5bc71bd229e6"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "3d2120dbb8e9faee538508edd1e0e09b"
server_key_iv: "da44f7cf576a740b1284b7b26164cb98"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.046944 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "001d82435f004ef4813e5bc71bd229e6"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "3d2120dbb8e9faee538508edd1e0e09b"
server_key_iv: "da44f7cf576a740b1284b7b26164cb98"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.050711 32170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:16.053249 32185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.054509 32170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:16.054654 32170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "001d82435f004ef4813e5bc71bd229e6"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "3d2120dbb8e9faee538508edd1e0e09b"
server_key_iv: "da44f7cf576a740b1284b7b26164cb98"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.054781 32170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.096341 32170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.099653 32170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.099884 32170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.100519 32170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:16.101404 32170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:16.101487 32170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.101581 32170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:16.101632 32170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.110852 32170 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:36397
I20260928 02:46:16.110893 32298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:36397 every 8 connection(s)
I20260928 02:46:16.111810 32170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:16.117295 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 32170
I20260928 02:46:16.117446 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:16.117745 29666 external_mini_cluster.cc:1428] Setting key 170b0af192c3d0c479af22c7fbcacab1
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:16.127784 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.113836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:57915 (local address 127.28.248.190:42215)
0928 02:46:16.113994 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.113999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.114964 (+ 965us) server_negotiation.cc:407] Connection header received
0928 02:46:16.116213 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.116216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.116266 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.116347 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.118022 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.118676 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.120311 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.120552 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.120733 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.123559 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.123565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.123572 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.123607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.125508 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.126049 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.126054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.126057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.126151 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.126490 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.126494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.126495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.126754 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.126878 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.127387 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.127558 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:16.128568 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.114294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:42215 (local address 127.28.248.129:57915)
0928 02:46:16.114775 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.114822 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.116009 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.116530 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.116538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.117040 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:16.117809 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.117836 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.118806 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.118811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.120182 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.120196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.120779 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.120786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.121046 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.121729 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:16.121745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:16.123331 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:16.125634 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.125639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.125653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.125944 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.126247 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.126249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.126251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.126389 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.126860 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:16.126865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:16.127135 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.127303 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.127583 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":107,"threads_started":1}
I20260928 02:46:16.130044 32299 heartbeater.cc:347] Connected to a master server at 127.28.248.190:42215
I20260928 02:46:16.130327 32299 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:16.130959 32299 heartbeater.cc:511] Master 127.28.248.190:42215 requested a full tablet report, sending...
I20260928 02:46:16.132694 32108 ts_manager.cc:194] Registered new tserver with Master: 001d82435f004ef4813e5bc71bd229e6 (127.28.248.129:36397)
I20260928 02:46:16.133852 32108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:57915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:16.148826 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:42215
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.262650 32304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.262949 32304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.263028 32304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.267160 32304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.267232 32304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.267329 32304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:16.272794 32304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:16.273964 32304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.275040 32304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.284746 32310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.284947 32304 server_base.cc:1034] running on GCE node
W20260928 02:46:16.284838 32312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.284746 32309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.285523 32304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.286166 32304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.287346 32304 hybrid_clock.cc:648] HybridClock initialized: now 1790563576287310 us; error 44 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.290455 32304 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:16.291358 32304 webserver.cc:460] Webserver started at http://127.28.248.130:45045/ using document root <none> and password file <none>
I20260928 02:46:16.291934 32304 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.292022 32304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.292250 32304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.294396 32304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "1d63c40dd0174e72b77327f94c31271e"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "d93ad064a01b2782ec61185bf9e91e05"
server_key_iv: "e475379871559ca8af662502e72ae6a0"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.294992 32304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "1d63c40dd0174e72b77327f94c31271e"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "d93ad064a01b2782ec61185bf9e91e05"
server_key_iv: "e475379871559ca8af662502e72ae6a0"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.298805 32304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:16.301391 32319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.302624 32304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:16.302758 32304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "1d63c40dd0174e72b77327f94c31271e"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "d93ad064a01b2782ec61185bf9e91e05"
server_key_iv: "e475379871559ca8af662502e72ae6a0"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.302865 32304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.322897 32304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.326090 32304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.326308 32304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.326929 32304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:16.327790 32304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:16.327870 32304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.327948 32304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:16.328002 32304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.337005 32304 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:43241
I20260928 02:46:16.337042 32432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:43241 every 8 connection(s)
I20260928 02:46:16.337975 32304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:16.345453 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 32304
I20260928 02:46:16.345546 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:16.345861 29666 external_mini_cluster.cc:1428] Setting key f310fa4e8a310da8c64b3271d3c3342f
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:16.352753 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.340083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:41229 (local address 127.28.248.190:42215)
0928 02:46:16.340242 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.340247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.341336 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 02:46:16.342682 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.342685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.342734 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.342812 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.343947 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.344461 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.345749 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.346085 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.346198 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.349012 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.349016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.349019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.349045 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.350561 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.351121 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.351124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.351126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.351213 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.351578 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.351581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.351583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.351777 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.351926 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.352439 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.352564 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:46:16.353668 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.340641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:42215 (local address 127.28.248.130:41229)
0928 02:46:16.341120 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.341182 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.342479 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.342955 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.342962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.343331 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:16.343818 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.343832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.344572 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.344574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.345630 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.345640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.346232 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.346252 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.346503 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.347125 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:16.347148 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:16.348842 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:16.350699 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.350706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.350723 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.351004 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.351308 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.351313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.351315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.351472 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.351922 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:16.351928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:16.352192 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.352388 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.352804 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260928 02:46:16.354897 32433 heartbeater.cc:347] Connected to a master server at 127.28.248.190:42215
I20260928 02:46:16.355175 32433 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:16.355722 32433 heartbeater.cc:511] Master 127.28.248.190:42215 requested a full tablet report, sending...
I20260928 02:46:16.356885 32108 ts_manager.cc:194] Registered new tserver with Master: 1d63c40dd0174e72b77327f94c31271e (127.28.248.130:43241)
I20260928 02:46:16.357489 32108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:41229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:16.377409 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:42215
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.490521 32438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.490790 32438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.490878 32438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.494932 32438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.494999 32438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.495100 32438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:16.500396 32438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:16.501562 32438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.502637 32438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.512243 32446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.512337 32438 server_base.cc:1034] running on GCE node
W20260928 02:46:16.512255 32443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.512300 32444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.512979 32438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.513648 32438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.514811 32438 hybrid_clock.cc:648] HybridClock initialized: now 1790563576514792 us; error 31 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.517562 32438 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:16.518373 32438 webserver.cc:460] Webserver started at http://127.28.248.131:43665/ using document root <none> and password file <none>
I20260928 02:46:16.518903 32438 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.518961 32438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.519125 32438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.521245 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "9a246f4af6e04d1887a4dfa6a68b11f2"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "e1e88421ec89eef790402ee411446469"
server_key_iv: "2c247318851d9df3c91601143a890184"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.521860 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "9a246f4af6e04d1887a4dfa6a68b11f2"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "e1e88421ec89eef790402ee411446469"
server_key_iv: "2c247318851d9df3c91601143a890184"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.525775 32438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:46:16.528221 32453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.529489 32438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:16.529624 32438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "9a246f4af6e04d1887a4dfa6a68b11f2"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-r238"
server_key: "e1e88421ec89eef790402ee411446469"
server_key_iv: "2c247318851d9df3c91601143a890184"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.529737 32438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.552820 32438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.555876 32438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.556089 32438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.556787 32438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:16.557634 32438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:16.557680 32438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.557750 32438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:16.557790 32438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.567178 32438 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:33717
I20260928 02:46:16.567194 32566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:33717 every 8 connection(s)
I20260928 02:46:16.568128 32438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:16.574097 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 32438
I20260928 02:46:16.574235 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:16.574568 29666 external_mini_cluster.cc:1428] Setting key cbc2ae0bc6a3c4ddba6a04ce3b6e4e43
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:16.582039 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.569930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:60539 (local address 127.28.248.190:42215)
0928 02:46:16.570081 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.570086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.571006 (+ 920us) server_negotiation.cc:407] Connection header received
0928 02:46:16.572227 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.572230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.572276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.572350 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.573737 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.574560 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.576071 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.576288 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.576369 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.578965 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.578968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.578970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.578994 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.580224 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.580753 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.580756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.580757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.580828 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.581127 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.581129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.581130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.581299 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.581382 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.581828 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.581922 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:16.582846 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.570385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:42215 (local address 127.28.248.131:60539)
0928 02:46:16.570821 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.570867 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.572037 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.572581 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.572588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.573070 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:16.573563 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.573578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.574747 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.574752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.575944 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.575954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.576439 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.576444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.576691 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.577360 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:16.577378 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:16.578815 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:16.580324 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.580330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.580341 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.580662 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.580907 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.580909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.580911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.581044 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.581379 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:16.581384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:16.581622 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.581772 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.582021 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260928 02:46:16.583909 32567 heartbeater.cc:347] Connected to a master server at 127.28.248.190:42215
I20260928 02:46:16.584157 32567 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:16.584692 32567 heartbeater.cc:511] Master 127.28.248.190:42215 requested a full tablet report, sending...
I20260928 02:46:16.585697 32108 ts_manager.cc:194] Registered new tserver with Master: 9a246f4af6e04d1887a4dfa6a68b11f2 (127.28.248.131:33717)
I20260928 02:46:16.586282 32108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:60539
I20260928 02:46:16.588598 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:46:16.601451 32147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:16.600040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.28.248.190:42215)
0928 02:46:16.600173 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.600177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.600313 (+ 136us) server_negotiation.cc:407] Connection header received
0928 02:46:16.600445 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.600448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.600490 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.600570 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.601319 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37108: BlockingRecv error: recv got EOF from 127.0.0.1:37108 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:46:16.602932 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 32170
I20260928 02:46:16.608979 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 32304
I20260928 02:46:16.614833 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 32438
I20260928 02:46:16.620481 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 32076
2026-09-28T02:46:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2201 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32579](info): set up 2 sockets
Sep 28 02:46:16 dist-test-slave-r238 krb5kdc[32579](info): commencing operation
krb5kdc: starting...
W20260928 02:46:17.672746 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:17 dist-test-slave-r238 krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:17.791630 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:40369
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:17.899255 32592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.899571 32592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.899657 32592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.903604 32592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:17.903692 32592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.903731 32592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.903795 32592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:17.903833 32592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:17.908828 32592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:17.909802 32592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.910771 32592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.918954 32598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.918982 32597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.919003 32600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.919289 32592 server_base.cc:1034] running on GCE node
I20260928 02:46:17.919822 32592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.920931 32592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.922087 32592 hybrid_clock.cc:648] HybridClock initialized: now 1790563577922059 us; error 39 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-r238 krb5kdc[32579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.930513 32592 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:17.931425 32592 webserver.cc:460] Webserver started at http://127.28.248.190:38353/ using document root <none> and password file <none>
I20260928 02:46:17.931972 32592 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.932031 32592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.932262 32592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:17.934336 32592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "c3b81158438f4f579a2db3c2128cfd8c"
format_stamp: "Formatted at 2026-09-28 02:46:17 on dist-test-slave-r238"
server_key: "998f64a7095f0eece6728db907c0959d"
server_key_iv: "d84b6e074c545874aa3d622741981546"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.934895 32592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "c3b81158438f4f579a2db3c2128cfd8c"
format_stamp: "Formatted at 2026-09-28 02:46:17 on dist-test-slave-r238"
server_key: "998f64a7095f0eece6728db907c0959d"
server_key_iv: "d84b6e074c545874aa3d622741981546"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.938530 32592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:17.940846 32607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.942028 32592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:17.942162 32592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "c3b81158438f4f579a2db3c2128cfd8c"
format_stamp: "Formatted at 2026-09-28 02:46:17 on dist-test-slave-r238"
server_key: "998f64a7095f0eece6728db907c0959d"
server_key_iv: "d84b6e074c545874aa3d622741981546"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.942267 32592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.959095 32592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.962347 32592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.962553 32592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.969556 32592 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:40369
I20260928 02:46:17.969830 32659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:40369 every 8 connection(s)
I20260928 02:46:17.970595 32592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:17.973245 32660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:17.977692 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 32592
I20260928 02:46:17.977819 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:17.978128 29666 external_mini_cluster.cc:1428] Setting key b3a54e8d237524c6cc58a7932deabfb7
I20260928 02:46:17.978981 32660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c: Bootstrap starting.
I20260928 02:46:17.981282 32660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:17.982003 32660 log.cc:824] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:17.983747 32660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c: No bootstrap required, opened a new log
Sep 28 02:46:17 dist-test-slave-r238 krb5kdc[32579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:17.987354 32660 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b81158438f4f579a2db3c2128cfd8c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40369 } }
I20260928 02:46:17.987612 32660 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:17.987696 32660 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b81158438f4f579a2db3c2128cfd8c, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.988253 32660 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b81158438f4f579a2db3c2128cfd8c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40369 } }
I20260928 02:46:17.988418 32660 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:17.988517 32660 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:17.988659 32660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:17.989517 32660 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b81158438f4f579a2db3c2128cfd8c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40369 } }
I20260928 02:46:17.989923 32660 leader_election.cc:302] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3b81158438f4f579a2db3c2128cfd8c; no voters:
I20260928 02:46:17.990252 32660 leader_election.cc:288] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:17.990321 32665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:17.990556 32665 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [term 1 LEADER]: Becoming Leader. State: Replica: c3b81158438f4f579a2db3c2128cfd8c, State: Running, Role: LEADER
I20260928 02:46:17.990903 32665 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b81158438f4f579a2db3c2128cfd8c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40369 } }
I20260928 02:46:17.991524 32660 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:17.992545 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b81158438f4f579a2db3c2128cfd8c. Latest consensus state: current_term: 1 leader_uuid: "c3b81158438f4f579a2db3c2128cfd8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b81158438f4f579a2db3c2128cfd8c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40369 } } }
I20260928 02:46:17.992681 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:17.992966 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.979425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.28.248.190:40369)
0928 02:46:17.979836 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.979850 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.979891 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:46:17.980506 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.980522 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.980916 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.981242 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.982046 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.983044 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.983938 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.984394 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.984566 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.986979 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.986984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.986996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.987019 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.989065 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.989503 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.989509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.989515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.989657 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.989972 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.989977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.989979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.990546 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.991400 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.991550 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.991809 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260928 02:46:17.993072 32666 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3b81158438f4f579a2db3c2128cfd8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b81158438f4f579a2db3c2128cfd8c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40369 } } }
I20260928 02:46:17.993268 32666 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:17.996196 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 02:46:17.996526 32680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:17.996577 32680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:17.996963 32674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:17.997588 32674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:18.003710 32674 catalog_manager.cc:1347] Generated new cluster ID: 3fdaaaa5f51d4bada84243617627a8d1
I20260928 02:46:18.003850 32674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:18.017593 32674 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:18.018782 32674 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:18.020112 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.014495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.28.248.190:40369)
0928 02:46:18.014611 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:18.014616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:18.014857 (+ 241us) server_negotiation.cc:407] Connection header received
0928 02:46:18.015032 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:18.015035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:18.015096 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:18.015194 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:18.015771 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:18.016595 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:18.017591 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:18.017949 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:18.018051 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:18.019038 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:18.019042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:18.019045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:18.019597 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:18.019629 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:18.019649 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:18.019733 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:18.019895 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:46:18.019969 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:46:18.025552 32674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3b81158438f4f579a2db3c2128cfd8c: Generated new TSK 0
I20260928 02:46:18.026294 32674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:18.034165 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 32592
2026-09-28T02:46:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1409 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:18 dist-test-slave-r238 krb5kdc[32693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:18 dist-test-slave-r238 krb5kdc[32693](info): set up 2 sockets
Sep 28 02:46:18 dist-test-slave-r238 krb5kdc[32693](info): commencing operation
krb5kdc: starting...
W20260928 02:46:19.080792 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:19.208006 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:35541
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:35541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:19.319628 32706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:19.319984 32706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:19.320070 32706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:19.324313 32706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:19.324436 32706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:19.324477 32706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:19.324515 32706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:19.324548 32706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:19.330322 32706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:35541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:35541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:19.331508 32706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:19.332571 32706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:19.341425 32712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:19.341483 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:19.341428 32711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:19.341456 32706 server_base.cc:1034] running on GCE node
I20260928 02:46:19.342320 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:19.343520 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:19.344686 32706 hybrid_clock.cc:648] HybridClock initialized: now 1790563579344676 us; error 35 us; skew 500 ppm
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:19.347500 32706 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:19.348521 32706 webserver.cc:460] Webserver started at http://127.28.248.190:36585/ using document root <none> and password file <none>
I20260928 02:46:19.349151 32706 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:19.349246 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:19.349485 32706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:19.351596 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "c4e98deb16094a7a8b7752d33a8d2da6"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "918758f0cac8e4b667b0ba4fdd0edb13"
server_key_iv: "6f8523875a0c41eb996551d8da3e7773"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.352177 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "c4e98deb16094a7a8b7752d33a8d2da6"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "918758f0cac8e4b667b0ba4fdd0edb13"
server_key_iv: "6f8523875a0c41eb996551d8da3e7773"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.356158 32706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:19.358855 32721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.360291 32706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:19.360471 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "c4e98deb16094a7a8b7752d33a8d2da6"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "918758f0cac8e4b667b0ba4fdd0edb13"
server_key_iv: "6f8523875a0c41eb996551d8da3e7773"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.360658 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:19.394546 32706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:19.397747 32706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:19.397969 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:19.405208 32706 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:35541
I20260928 02:46:19.405501 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:35541 every 8 connection(s)
I20260928 02:46:19.406281 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:19.409149 306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:19.414927 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6: Bootstrap starting.
I20260928 02:46:19.415059 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 32706
I20260928 02:46:19.415160 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:19.415450 29666 external_mini_cluster.cc:1428] Setting key bbad72dae0e2ce9c4d9a9065f724f139
I20260928 02:46:19.417766 306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:19.418524 306 log.cc:824] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:19.420455 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6: No bootstrap required, opened a new log
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:19.424472 306 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35541 } }
I20260928 02:46:19.424911 306 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:19.425087 306 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e98deb16094a7a8b7752d33a8d2da6, State: Initialized, Role: FOLLOWER
I20260928 02:46:19.425725 306 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35541 } }
I20260928 02:46:19.425907 306 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:19.426013 306 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:19.426117 306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:19.427131 306 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35541 } }
I20260928 02:46:19.427501 306 leader_election.cc:302] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e98deb16094a7a8b7752d33a8d2da6; no voters:
I20260928 02:46:19.427847 306 leader_election.cc:288] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:19.428182 311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:19.428556 311 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [term 1 LEADER]: Becoming Leader. State: Replica: c4e98deb16094a7a8b7752d33a8d2da6, State: Running, Role: LEADER
I20260928 02:46:19.428994 311 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35541 } }
I20260928 02:46:19.429273 306 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:19.430532 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.416890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.28.248.190:35541)
0928 02:46:19.417355 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:19.417371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:19.417406 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:46:19.418121 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:19.418140 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:19.418488 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:19.418893 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:19.419685 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.420807 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.421673 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.422159 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.422326 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.425240 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:19.425245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:19.425260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:19.425290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:19.427087 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.427570 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.427576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.427581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.427754 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.428048 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.428053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.428055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.428683 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:19.428922 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:19.429228 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 02:46:19.429552 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":148,"threads_started":1}
I20260928 02:46:19.430795 312 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35541 } } }
I20260928 02:46:19.430948 312 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:19.431273 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e98deb16094a7a8b7752d33a8d2da6. Latest consensus state: current_term: 1 leader_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e98deb16094a7a8b7752d33a8d2da6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 35541 } } }
I20260928 02:46:19.431349 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6 [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:19.434415 326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:19.434504 326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:19.434559 323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:19.435442 323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:19.441392 323 catalog_manager.cc:1347] Generated new cluster ID: b256c31fb02545288e3d4ccaf00f66c4
I20260928 02:46:19.441468 323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:19.458940 323 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:19.459921 323 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:19.469398 323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e98deb16094a7a8b7752d33a8d2da6: Generated new TSK 0
I20260928 02:46:19.470160 323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:19.509294 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35541
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:19.626958 332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:19.627244 332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:19.627290 332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:19.631803 332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:19.631882 332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:19.631978 332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:19.637424 332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:19.638602 332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:19.639722 332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:19.649688 337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:19.649645 340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:19.649645 338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:19.650015 332 server_base.cc:1034] running on GCE node
I20260928 02:46:19.650729 332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:19.651440 332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:19.652678 332 hybrid_clock.cc:648] HybridClock initialized: now 1790563579652654 us; error 27 us; skew 500 ppm
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:19.655534 332 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:19.656424 332 webserver.cc:460] Webserver started at http://127.28.248.129:38765/ using document root <none> and password file <none>
I20260928 02:46:19.657008 332 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:19.657069 332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:19.657229 332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:19.659250 332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "bd976924b08d454d96eea84ffb396708"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "a60675439204ff734c3a317377116e9c"
server_key_iv: "9d667e53d1183a628ca077ff8b0ea39e"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.659790 332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "bd976924b08d454d96eea84ffb396708"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "a60675439204ff734c3a317377116e9c"
server_key_iv: "9d667e53d1183a628ca077ff8b0ea39e"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.663667 332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:19.666445 347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.668043 332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:19.668164 332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "bd976924b08d454d96eea84ffb396708"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "a60675439204ff734c3a317377116e9c"
server_key_iv: "9d667e53d1183a628ca077ff8b0ea39e"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.668267 332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:19.697966 332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:19.701553 332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:19.701725 332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:19.702338 332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:19.703138 332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:19.703186 332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.703227 332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:19.703250 332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.712424 332 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:46677
I20260928 02:46:19.712448 460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:46677 every 8 connection(s)
I20260928 02:46:19.713442 332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:19.717025 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 332
I20260928 02:46:19.717167 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:19.717531 29666 external_mini_cluster.cc:1428] Setting key 8c2c5f69b82ed55966101b595d3b44b6
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:19.729388 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.715438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:49511 (local address 127.28.248.190:35541)
0928 02:46:19.715584 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:19.715588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:19.716504 (+ 916us) server_negotiation.cc:407] Connection header received
0928 02:46:19.718379 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:19.718382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:19.718426 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:19.718497 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:19.720148 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.720828 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.721992 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.722461 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.722587 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.725443 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:19.725447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:19.725451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:19.725482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:19.727183 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.727724 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.727729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.727733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.727825 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.728175 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.728177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.728178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.728362 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:19.728450 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:19.729143 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 02:46:19.729257 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:19.730362 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.715890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:35541 (local address 127.28.248.129:49511)
0928 02:46:19.716343 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.716388 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:19.718147 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:19.718717 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:19.718727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:19.719216 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:19.719972 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.719996 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.720957 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.720960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.721859 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.721868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.722613 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.722616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.722759 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.723436 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:19.723453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:19.725238 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:19.727305 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.727310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.727323 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.727594 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.727936 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.727938 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.727939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.728087 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.728462 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:19.728467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:19.728827 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 02:46:19.729102 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 02:46:19.729406 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:19.731863 461 heartbeater.cc:347] Connected to a master server at 127.28.248.190:35541
I20260928 02:46:19.732129 461 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:19.732853 461 heartbeater.cc:511] Master 127.28.248.190:35541 requested a full tablet report, sending...
I20260928 02:46:19.734669 32738 ts_manager.cc:194] Registered new tserver with Master: bd976924b08d454d96eea84ffb396708 (127.28.248.129:46677)
I20260928 02:46:19.736433 32738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:49511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:19.747995 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35541
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:19.863468 466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:19.863775 466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:19.863818 466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:19.868126 466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:19.868209 466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:19.868302 466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:19.873908 466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:19.875059 466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:19.876159 466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:19.886140 474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:19.886262 471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:19.886358 466 server_base.cc:1034] running on GCE node
W20260928 02:46:19.886620 472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:19.887024 466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:19.887696 466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:19.888903 466 hybrid_clock.cc:648] HybridClock initialized: now 1790563579888864 us; error 31 us; skew 500 ppm
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:19.891601 466 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:19.892434 466 webserver.cc:460] Webserver started at http://127.28.248.130:34387/ using document root <none> and password file <none>
I20260928 02:46:19.893040 466 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:19.893103 466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:19.893276 466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:19.895438 466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "7d5f3cb2ebcb4f9a8dccb7ccfb3b51ef"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "f4c6cfd64c64893f19bc941bad6c9fa8"
server_key_iv: "08a0d25bc11a3717329674b58563ffa7"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.895967 466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "7d5f3cb2ebcb4f9a8dccb7ccfb3b51ef"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "f4c6cfd64c64893f19bc941bad6c9fa8"
server_key_iv: "08a0d25bc11a3717329674b58563ffa7"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.899747 466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:19.902246 481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.903488 466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:19.903616 466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "7d5f3cb2ebcb4f9a8dccb7ccfb3b51ef"
format_stamp: "Formatted at 2026-09-28 02:46:19 on dist-test-slave-r238"
server_key: "f4c6cfd64c64893f19bc941bad6c9fa8"
server_key_iv: "08a0d25bc11a3717329674b58563ffa7"
server_key_version: "encryptionkey@0"
I20260928 02:46:19.903728 466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:19.920295 466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:19.923412 466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:19.923624 466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:19.924252 466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:19.925165 466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:19.925213 466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.925280 466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:19.925320 466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:19.935003 466 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:41061
I20260928 02:46:19.935022 594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:41061 every 8 connection(s)
I20260928 02:46:19.935921 466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:19.945259 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 466
I20260928 02:46:19.945394 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
Sep 28 02:46:19 dist-test-slave-r238 krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:19.945708 29666 external_mini_cluster.cc:1428] Setting key deece5fc664ea3153396be318746b582
I20260928 02:46:19.949913 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.937691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:42035 (local address 127.28.248.190:35541)
0928 02:46:19.937825 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:19.937829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:19.938776 (+ 947us) server_negotiation.cc:407] Connection header received
0928 02:46:19.940197 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:19.940200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:19.940258 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:19.940359 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:19.941601 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.942134 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.943276 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.943501 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.943609 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.946355 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:19.946358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:19.946360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:19.946383 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:19.947827 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.948347 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.948352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.948354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.948439 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.948841 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.948846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.948848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.949023 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:19.949131 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:19.949634 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 02:46:19.949749 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:46:19.950814 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.938150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:35541 (local address 127.28.248.130:42035)
0928 02:46:19.938617 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.938662 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:19.939976 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:19.940549 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:19.940555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:19.940976 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:19.941456 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.941469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.942261 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.942264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.943150 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.943159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.943650 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.943654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.943826 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.944395 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:19.944411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:19.946179 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:19.947965 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.947971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.947985 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.948240 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.948533 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.948538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.948540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.948747 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.949127 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:19.949133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:19.949393 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 02:46:19.949577 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:46:19.949899 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260928 02:46:19.951926 595 heartbeater.cc:347] Connected to a master server at 127.28.248.190:35541
I20260928 02:46:19.952175 595 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:19.952709 595 heartbeater.cc:511] Master 127.28.248.190:35541 requested a full tablet report, sending...
I20260928 02:46:19.953760 32738 ts_manager.cc:194] Registered new tserver with Master: 7d5f3cb2ebcb4f9a8dccb7ccfb3b51ef (127.28.248.130:41061)
I20260928 02:46:19.954588 32738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:42035
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:19.971874 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35541
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:20.083452 600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:20.083757 600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:20.083835 600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:20.087883 600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:20.087988 600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:20.088085 600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:20.093385 600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:35541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:20.094525 600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:20.095757 600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:20.105526 605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:20.105522 606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:20.105633 600 server_base.cc:1034] running on GCE node
W20260928 02:46:20.105545 608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:20.106284 600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:20.107056 600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:20.108246 600 hybrid_clock.cc:648] HybridClock initialized: now 1790563580108209 us; error 50 us; skew 500 ppm
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563580, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:20.111016 600 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:20.111868 600 webserver.cc:460] Webserver started at http://127.28.248.131:38515/ using document root <none> and password file <none>
I20260928 02:46:20.112449 600 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:20.112541 600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:20.112825 600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:20.114876 600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "25d91b0fdceb451fac65a14f649ac7d9"
format_stamp: "Formatted at 2026-09-28 02:46:20 on dist-test-slave-r238"
server_key: "533d53a00ab7fdf1e9785a5112560b07"
server_key_iv: "72f96c3f243f5005e9f18415baab96cb"
server_key_version: "encryptionkey@0"
I20260928 02:46:20.115454 600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "25d91b0fdceb451fac65a14f649ac7d9"
format_stamp: "Formatted at 2026-09-28 02:46:20 on dist-test-slave-r238"
server_key: "533d53a00ab7fdf1e9785a5112560b07"
server_key_iv: "72f96c3f243f5005e9f18415baab96cb"
server_key_version: "encryptionkey@0"
I20260928 02:46:20.119302 600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:20.121748 615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:20.122973 600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:20.123107 600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "25d91b0fdceb451fac65a14f649ac7d9"
format_stamp: "Formatted at 2026-09-28 02:46:20 on dist-test-slave-r238"
server_key: "533d53a00ab7fdf1e9785a5112560b07"
server_key_iv: "72f96c3f243f5005e9f18415baab96cb"
server_key_version: "encryptionkey@0"
I20260928 02:46:20.123219 600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:20.142089 600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:20.145040 600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:20.145241 600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:20.145830 600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:20.146649 600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:20.146721 600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:20.146790 600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:20.146840 600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:20.155743 600 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:41915
I20260928 02:46:20.155821 728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:41915 every 8 connection(s)
I20260928 02:46:20.156771 600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:20.158270 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 600
I20260928 02:46:20.158397 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:20.158723 29666 external_mini_cluster.cc:1428] Setting key 7917798a209dd7dbc352707b387c212d
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563580, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:20.170789 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:20.158717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:44983 (local address 127.28.248.190:35541)
0928 02:46:20.158877 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:20.158881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:20.159857 (+ 976us) server_negotiation.cc:407] Connection header received
0928 02:46:20.161448 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:20.161451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:20.161504 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:20.161581 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:20.162777 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:20.163316 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:20.164543 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:20.164853 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:20.164997 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:20.167458 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:20.167461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:20.167463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:20.167487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:20.169048 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:20.169500 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:20.169502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:20.169504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:20.169575 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:20.169858 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:20.169860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:20.169861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:20.170029 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:20.170101 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:20.170563 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 02:46:20.170658 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:20.171737 731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:20.159203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:35541 (local address 127.28.248.131:44983)
0928 02:46:20.159700 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:20.159761 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:20.161238 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:20.161756 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:20.161763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:20.162124 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:20.162628 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:20.162641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:20.163449 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:20.163453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:20.164415 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:20.164425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:20.165039 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:20.165043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:20.165221 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:20.165818 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:20.165834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:20.167302 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:20.169142 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:20.169147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:20.169161 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:20.169413 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:20.169653 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:20.169655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:20.169656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:20.169782 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:20.170112 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:20.170117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:20.170344 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 02:46:20.170519 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:46:20.170782 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
I20260928 02:46:20.173013 729 heartbeater.cc:347] Connected to a master server at 127.28.248.190:35541
I20260928 02:46:20.173269 729 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:20.173759 729 heartbeater.cc:511] Master 127.28.248.190:35541 requested a full tablet report, sending...
I20260928 02:46:20.174791 32738 ts_manager.cc:194] Registered new tserver with Master: 25d91b0fdceb451fac65a14f649ac7d9 (127.28.248.131:41915)
I20260928 02:46:20.175356 32738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:44983
I20260928 02:46:20.184510 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563580, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563580, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:20.210242 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:20.201263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.28.248.190:35541)
0928 02:46:20.201441 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:20.201446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:20.201596 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:46:20.201843 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:20.201846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:20.201892 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:20.201999 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:20.202769 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:20.203317 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:20.204185 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:20.204417 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:20.204538 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:20.207159 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:20.207162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:20.207165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:20.207188 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:20.208524 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:20.208983 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:20.208992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:20.208993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:20.209066 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:20.209347 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:20.209351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:20.209353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:20.209583 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:20.209723 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:20.209955 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:46:20.210075 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:46:20.210661 32738 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51050
I20260928 02:46:20.211648 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 332
I20260928 02:46:20.217109 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 466
I20260928 02:46:20.221851 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 600
I20260928 02:46:20.226908 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 32706
2026-09-28T02:46:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2195 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[741](info): set up 2 sockets
Sep 28 02:46:20 dist-test-slave-r238 krb5kdc[741](info): commencing operation
krb5kdc: starting...
W20260928 02:46:21.264827 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:21 dist-test-slave-r238 krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:21.392835 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46699
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:46699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:21.518462 754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:21.518786 754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:21.518832 754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:21.523046 754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:21.523129 754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:21.523152 754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:21.523173 754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:21.523191 754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:21.528846 754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:46699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:21.530025 754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:21.531157 754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:21.540341 760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:21.540349 759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:21.540544 754 server_base.cc:1034] running on GCE node
W20260928 02:46:21.540359 762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:21.541375 754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:21.542591 754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:21.543777 754 hybrid_clock.cc:648] HybridClock initialized: now 1790563581543754 us; error 42 us; skew 500 ppm
Sep 28 02:46:21 dist-test-slave-r238 krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:21.546674 754 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:21.547706 754 webserver.cc:460] Webserver started at http://127.28.248.190:41611/ using document root <none> and password file <none>
I20260928 02:46:21.548313 754 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:21.548406 754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:21.548712 754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:21.550865 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "a57ad2747a9d4ebfb750d63bddcba6a1"
format_stamp: "Formatted at 2026-09-28 02:46:21 on dist-test-slave-r238"
server_key: "64d18ab8e36c341e6bc8c9ec9dcce57c"
server_key_iv: "640183467b65784f1fe92b99aecc5d77"
server_key_version: "encryptionkey@0"
I20260928 02:46:21.551473 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "a57ad2747a9d4ebfb750d63bddcba6a1"
format_stamp: "Formatted at 2026-09-28 02:46:21 on dist-test-slave-r238"
server_key: "64d18ab8e36c341e6bc8c9ec9dcce57c"
server_key_iv: "640183467b65784f1fe92b99aecc5d77"
server_key_version: "encryptionkey@0"
I20260928 02:46:21.555394 754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:46:21.558045 769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:21.559312 754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:21.559473 754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "a57ad2747a9d4ebfb750d63bddcba6a1"
format_stamp: "Formatted at 2026-09-28 02:46:21 on dist-test-slave-r238"
server_key: "64d18ab8e36c341e6bc8c9ec9dcce57c"
server_key_iv: "640183467b65784f1fe92b99aecc5d77"
server_key_version: "encryptionkey@0"
I20260928 02:46:21.559598 754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:21.573046 754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:21.576102 754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:21.576323 754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:21.583992 754 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:46699
I20260928 02:46:21.584093 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:46699 every 8 connection(s)
I20260928 02:46:21.585079 754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:21.588070 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:21.593992 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 754
I20260928 02:46:21.594144 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:21.594388 822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1: Bootstrap starting.
I20260928 02:46:21.594466 29666 external_mini_cluster.cc:1428] Setting key 4efba092c9461e3441e2e3c6b7e6cf56
I20260928 02:46:21.597191 822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:21.598006 822 log.cc:824] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:21.600160 822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1: No bootstrap required, opened a new log
Sep 28 02:46:21 dist-test-slave-r238 krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:21.604504 822 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46699 } }
I20260928 02:46:21.604882 822 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:21.604952 822 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57ad2747a9d4ebfb750d63bddcba6a1, State: Initialized, Role: FOLLOWER
I20260928 02:46:21.605557 822 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46699 } }
I20260928 02:46:21.605774 822 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:21.605855 822 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:21.605968 822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:21.607008 822 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46699 } }
I20260928 02:46:21.607401 822 leader_election.cc:302] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57ad2747a9d4ebfb750d63bddcba6a1; no voters:
I20260928 02:46:21.607769 822 leader_election.cc:288] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:21.607893 827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:21.608152 827 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [term 1 LEADER]: Becoming Leader. State: Replica: a57ad2747a9d4ebfb750d63bddcba6a1, State: Running, Role: LEADER
I20260928 02:46:21.608590 827 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46699 } }
I20260928 02:46:21.609289 822 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:21.610695 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:21.595983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.28.248.190:46699)
0928 02:46:21.596457 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:21.596472 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:21.596509 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:46:21.597207 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:21.597224 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:21.597617 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:21.597968 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:21.598827 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:21.599896 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:21.600885 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:21.601368 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:21.601551 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:21.604419 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:21.604424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:21.604442 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:21.604472 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:21.606545 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:21.607017 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:21.607028 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:21.607034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:21.607207 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:21.607505 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:21.607508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:21.607510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:21.608126 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:21.609105 (+ 979us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:21.609259 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:46:21.609574 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":146,"threads_started":1}
I20260928 02:46:21.612362 828 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46699 } } }
I20260928 02:46:21.612499 828 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:21.609974 827 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57ad2747a9d4ebfb750d63bddcba6a1. Latest consensus state: current_term: 1 leader_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57ad2747a9d4ebfb750d63bddcba6a1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46699 } } }
I20260928 02:46:21.612777 827 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1 [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:21.614888 841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:21.614965 841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:21.615056 843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:21.615835 843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:21.622174 843 catalog_manager.cc:1347] Generated new cluster ID: ea48222ba7dd40b0aada6e7726d305d9
I20260928 02:46:21.622288 843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:21.629908 843 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:21.630980 843 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:21.641746 843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a57ad2747a9d4ebfb750d63bddcba6a1: Generated new TSK 0
I20260928 02:46:21.642591 843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:21.676389 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46699
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:21.787859 848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:21.788120 848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:21.788162 848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:21.792219 848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:21.792276 848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:21.792393 848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:21.797621 848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:21.798708 848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:21.799809 848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:21.809197 856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:21.809262 854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:21.809195 848 server_base.cc:1034] running on GCE node
W20260928 02:46:21.809190 853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:21.809947 848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:21.810534 848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:21.811687 848 hybrid_clock.cc:648] HybridClock initialized: now 1790563581811658 us; error 51 us; skew 500 ppm
Sep 28 02:46:21 dist-test-slave-r238 krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:21.814422 848 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:21.815263 848 webserver.cc:460] Webserver started at http://127.28.248.129:44449/ using document root <none> and password file <none>
I20260928 02:46:21.815809 848 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:21.815868 848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:21.816087 848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:21.818154 848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "b3f34e4e6bee487599d34ace1369cb5d"
format_stamp: "Formatted at 2026-09-28 02:46:21 on dist-test-slave-r238"
server_key: "2ca5a8612d7f8f37702bed7d412d9a06"
server_key_iv: "98ebe4e15210b601e3e25cbacf72a93b"
server_key_version: "encryptionkey@0"
I20260928 02:46:21.818706 848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "b3f34e4e6bee487599d34ace1369cb5d"
format_stamp: "Formatted at 2026-09-28 02:46:21 on dist-test-slave-r238"
server_key: "2ca5a8612d7f8f37702bed7d412d9a06"
server_key_iv: "98ebe4e15210b601e3e25cbacf72a93b"
server_key_version: "encryptionkey@0"
I20260928 02:46:21.822414 848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:46:21.824836 863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:21.826006 848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:21.826135 848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "b3f34e4e6bee487599d34ace1369cb5d"
format_stamp: "Formatted at 2026-09-28 02:46:21 on dist-test-slave-r238"
server_key: "2ca5a8612d7f8f37702bed7d412d9a06"
server_key_iv: "98ebe4e15210b601e3e25cbacf72a93b"
server_key_version: "encryptionkey@0"
I20260928 02:46:21.826241 848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:21.858987 848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:21.862275 848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:21.862494 848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:21.863224 848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:21.864177 848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:21.864256 848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:21.864333 848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:21.864368 848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:21.873180 848 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:44333
I20260928 02:46:21.873202 976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:44333 every 8 connection(s)
I20260928 02:46:21.874181 848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:21.883792 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 848
Sep 28 02:46:21 dist-test-slave-r238 krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:21.883919 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:21.884239 29666 external_mini_cluster.cc:1428] Setting key 068f824b0755a51d5a01c7576b07b02c
I20260928 02:46:21.888499 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:21.876047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:55583 (local address 127.28.248.190:46699)
0928 02:46:21.876188 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:21.876191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:21.877047 (+ 856us) server_negotiation.cc:407] Connection header received
0928 02:46:21.878405 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:21.878408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:21.878459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:21.878536 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:21.879716 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:21.880244 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:21.881423 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:21.881644 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:21.881722 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:21.884636 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:21.884640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:21.884652 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:21.884690 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:21.886314 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:21.886929 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:21.886932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:21.886933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:21.887000 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:21.887391 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:21.887393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:21.887394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:21.887588 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:21.887687 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:21.888202 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 02:46:21.888324 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:21.889357 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:21.876357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46699 (local address 127.28.248.129:55583)
0928 02:46:21.876856 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:21.876938 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:21.878199 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:21.878707 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:21.878714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:21.879108 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:21.879588 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:21.879602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:21.880352 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:21.880355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:21.881319 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:21.881328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:21.881775 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:21.881778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:21.881943 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:21.882579 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:21.882595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:21.884399 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:21.886428 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:21.886435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:21.886453 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:21.886829 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:21.887088 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:21.887094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:21.887099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:21.887286 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:21.887705 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:21.887712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:21.887968 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 02:46:21.888165 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:46:21.888498 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":108,"threads_started":1}
I20260928 02:46:21.890561 977 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46699
I20260928 02:46:21.890797 977 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:21.891283 977 heartbeater.cc:511] Master 127.28.248.190:46699 requested a full tablet report, sending...
I20260928 02:46:21.892812 786 ts_manager.cc:194] Registered new tserver with Master: b3f34e4e6bee487599d34ace1369cb5d (127.28.248.129:44333)
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:21.893967 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:55583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:21.912580 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46699
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:22.032555 982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:22.033004 982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:22.033088 982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:22.038830 982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:22.038904 982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:22.039003 982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:22.046751 982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:22.048278 982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:22.049454 982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:22.059122 990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:22.059118 987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:22.059345 982 server_base.cc:1034] running on GCE node
W20260928 02:46:22.059118 988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:22.059788 982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:22.060439 982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:22.061599 982 hybrid_clock.cc:648] HybridClock initialized: now 1790563582061553 us; error 60 us; skew 500 ppm
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:22.064535 982 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:22.065502 982 webserver.cc:460] Webserver started at http://127.28.248.130:38055/ using document root <none> and password file <none>
I20260928 02:46:22.066048 982 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:22.066107 982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:22.066272 982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:22.068333 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "81ae364c3c5b42efa230dda5d9aa5765"
format_stamp: "Formatted at 2026-09-28 02:46:22 on dist-test-slave-r238"
server_key: "c4c06ba755bf2346254874f173554023"
server_key_iv: "750f9a2a10db6f64af6046aeb023f247"
server_key_version: "encryptionkey@0"
I20260928 02:46:22.068957 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "81ae364c3c5b42efa230dda5d9aa5765"
format_stamp: "Formatted at 2026-09-28 02:46:22 on dist-test-slave-r238"
server_key: "c4c06ba755bf2346254874f173554023"
server_key_iv: "750f9a2a10db6f64af6046aeb023f247"
server_key_version: "encryptionkey@0"
I20260928 02:46:22.072721 982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:46:22.075166 997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:22.076418 982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:22.076560 982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "81ae364c3c5b42efa230dda5d9aa5765"
format_stamp: "Formatted at 2026-09-28 02:46:22 on dist-test-slave-r238"
server_key: "c4c06ba755bf2346254874f173554023"
server_key_iv: "750f9a2a10db6f64af6046aeb023f247"
server_key_version: "encryptionkey@0"
I20260928 02:46:22.076718 982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:22.089406 982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:22.092844 982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:22.093053 982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:22.093657 982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:22.094502 982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:22.094568 982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:22.094637 982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:22.094686 982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:22.103770 982 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:35625
I20260928 02:46:22.103788 1110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:35625 every 8 connection(s)
I20260928 02:46:22.104830 982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:22.109166 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 982
I20260928 02:46:22.109339 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:22.109606 29666 external_mini_cluster.cc:1428] Setting key eeea418d7f95096c0f625edb597f6a09
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:22.120839 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.106689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:38759 (local address 127.28.248.190:46699)
0928 02:46:22.106833 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:22.106837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:22.107645 (+ 808us) server_negotiation.cc:407] Connection header received
0928 02:46:22.109101 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:22.109104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:22.109180 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:22.109287 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:22.110845 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.111503 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.113293 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.113671 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.113793 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:22.116602 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:22.116636 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:22.116642 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:22.116678 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:22.118418 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.119010 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.119016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.119021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.119112 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.119482 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.119487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.119490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.119774 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:22.119889 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:22.120505 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 02:46:22.120696 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:46:22.121829 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.107070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46699 (local address 127.28.248.130:38759)
0928 02:46:22.107494 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:22.107537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:22.108864 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:22.109557 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:22.109563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:22.109965 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:22.110679 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:22.110699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.111642 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.111649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:22.113175 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:22.113185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.113825 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.113827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:22.114006 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:22.114624 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:22.114640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:22.116416 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:22.118571 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:22.118577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:22.118591 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:22.118894 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:22.119218 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:22.119221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:22.119222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:22.119377 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:22.119906 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:22.119912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:22.120227 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 02:46:22.120427 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 02:46:22.120777 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:46:22.123322 1111 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46699
I20260928 02:46:22.123608 1111 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:22.124181 1111 heartbeater.cc:511] Master 127.28.248.190:46699 requested a full tablet report, sending...
I20260928 02:46:22.125398 786 ts_manager.cc:194] Registered new tserver with Master: 81ae364c3c5b42efa230dda5d9aa5765 (127.28.248.130:35625)
I20260928 02:46:22.126341 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:38759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:22.140560 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46699
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:22.254151 1116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:22.254442 1116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:22.254487 1116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:22.259070 1116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:22.259131 1116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:22.259243 1116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:22.264775 1116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:22.265942 1116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:22.267091 1116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:22.276880 1121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:22.276880 1124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:22.276912 1122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:22.277186 1116 server_base.cc:1034] running on GCE node
I20260928 02:46:22.277606 1116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:22.278271 1116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:22.279446 1116 hybrid_clock.cc:648] HybridClock initialized: now 1790563582279414 us; error 48 us; skew 500 ppm
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:22.282219 1116 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:22.283098 1116 webserver.cc:460] Webserver started at http://127.28.248.131:44135/ using document root <none> and password file <none>
I20260928 02:46:22.283651 1116 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:22.283739 1116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:22.283963 1116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:22.286000 1116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "6697d9e35366499bb7946c42c4bf7796"
format_stamp: "Formatted at 2026-09-28 02:46:22 on dist-test-slave-r238"
server_key: "dcba4a22fd1ae15372721b134ca1119c"
server_key_iv: "15aa1a130efe3b6120906ed56678671f"
server_key_version: "encryptionkey@0"
I20260928 02:46:22.286562 1116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "6697d9e35366499bb7946c42c4bf7796"
format_stamp: "Formatted at 2026-09-28 02:46:22 on dist-test-slave-r238"
server_key: "dcba4a22fd1ae15372721b134ca1119c"
server_key_iv: "15aa1a130efe3b6120906ed56678671f"
server_key_version: "encryptionkey@0"
I20260928 02:46:22.290184 1116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:22.292506 1131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:22.293711 1116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:22.293841 1116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "6697d9e35366499bb7946c42c4bf7796"
format_stamp: "Formatted at 2026-09-28 02:46:22 on dist-test-slave-r238"
server_key: "dcba4a22fd1ae15372721b134ca1119c"
server_key_iv: "15aa1a130efe3b6120906ed56678671f"
server_key_version: "encryptionkey@0"
I20260928 02:46:22.293948 1116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:22.303843 1116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:22.307024 1116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:22.307226 1116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:22.307792 1116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:22.308580 1116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:22.308703 1116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:22.308774 1116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:22.308843 1116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:22.317600 1116 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:34123
I20260928 02:46:22.317625 1244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:34123 every 8 connection(s)
I20260928 02:46:22.318562 1116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:22.327795 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1116
I20260928 02:46:22.327955 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:22.328295 29666 external_mini_cluster.cc:1428] Setting key f6906008d730cb7958583139668b3bb6
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:22.332813 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.320330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:32789 (local address 127.28.248.190:46699)
0928 02:46:22.320480 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:22.320484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:22.321376 (+ 892us) server_negotiation.cc:407] Connection header received
0928 02:46:22.322765 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:22.322769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:22.322824 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:22.322930 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:22.324135 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.324817 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.325939 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.326155 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.326265 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:22.329184 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:22.329189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:22.329193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:22.329222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:22.330860 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.331365 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.331368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.331370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.331448 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.331740 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.331742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.331743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.331938 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:22.332040 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:22.332499 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 02:46:22.332592 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:46:22.333693 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.320802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46699 (local address 127.28.248.131:32789)
0928 02:46:22.321228 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:22.321269 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:22.322554 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:22.323134 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:22.323141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:22.323503 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:22.323985 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:22.323997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.324951 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.324954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:22.325804 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:22.325812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.326308 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.326312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:22.326479 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:22.327202 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:22.327219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:22.329005 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:22.331000 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:22.331005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:22.331018 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:22.331279 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:22.331527 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:22.331529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:22.331531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:22.331664 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:22.332036 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:22.332045 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:22.332287 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 02:46:22.332462 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:46:22.332863 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260928 02:46:22.334815 1245 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46699
I20260928 02:46:22.335055 1245 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:22.335500 1245 heartbeater.cc:511] Master 127.28.248.190:46699 requested a full tablet report, sending...
I20260928 02:46:22.336429 786 ts_manager.cc:194] Registered new tserver with Master: 6697d9e35366499bb7946c42c4bf7796 (127.28.248.131:34123)
I20260928 02:46:22.337039 786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:32789
I20260928 02:46:22.342298 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:22.368808 1252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.359152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.28.248.129:44333)
0928 02:46:22.359486 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:22.359500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:22.359528 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:46:22.359580 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:22.359585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:22.359779 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:22.359900 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:22.360538 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.361389 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.362198 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.362563 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.362644 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:22.365297 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:22.365306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:22.365320 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:22.365346 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:22.367115 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.367527 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.367532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.367534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.367613 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.367847 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.367849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.367850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.368111 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:22.368287 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:22.368427 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:46:22.368557 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:22.379757 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.371872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.28.248.190:46699)
0928 02:46:22.372020 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:22.372024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:22.372038 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:22.372135 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:22.372137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:22.372179 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:22.372246 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:22.372945 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.373454 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.374212 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:22.374425 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:22.374501 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:22.377006 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:22.377009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:22.377011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:22.377039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:22.378301 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.378675 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.378678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.378679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.378748 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:22.378969 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:22.378971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:22.378972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:22.379187 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:22.379288 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:22.379479 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:46:22.379614 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:46:22.380190 786 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59068
I20260928 02:46:22.380887 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 848
I20260928 02:46:22.386549 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 982
I20260928 02:46:22.391837 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1116
I20260928 02:46:22.397130 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 754
2026-09-28T02:46:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2170 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[1257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[1257](info): set up 2 sockets
Sep 28 02:46:22 dist-test-slave-r238 krb5kdc[1257](info): commencing operation
krb5kdc: starting...
W20260928 02:46:23.437083 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:23 dist-test-slave-r238 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:23.564013 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39063
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:23.677096 1270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:23.677448 1270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:23.677541 1270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:23.681730 1270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:23.681864 1270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:23.681900 1270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:23.681942 1270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:23.681972 1270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:23.691184 1270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:23.692289 1270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:23.693380 1270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:23.701572 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:23.701565 1275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:23.701577 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:23.701924 1270 server_base.cc:1034] running on GCE node
I20260928 02:46:23.702544 1270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:23.703692 1270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:23.704891 1270 hybrid_clock.cc:648] HybridClock initialized: now 1790563583704873 us; error 33 us; skew 500 ppm
Sep 28 02:46:23 dist-test-slave-r238 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:23.711187 1270 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:23.711968 1270 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:23.712034 1270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:23.712208 1270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:23.714339 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a"
format_stamp: "Formatted at 2026-09-28 02:46:23 on dist-test-slave-r238"
server_key: "d571e573a08a6eb360c188d98c0e36ee"
server_key_iv: "2f95be5b22205cfe2571270f8a2afd13"
server_key_version: "encryptionkey@0"
I20260928 02:46:23.714877 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a"
format_stamp: "Formatted at 2026-09-28 02:46:23 on dist-test-slave-r238"
server_key: "d571e573a08a6eb360c188d98c0e36ee"
server_key_iv: "2f95be5b22205cfe2571270f8a2afd13"
server_key_version: "encryptionkey@0"
I20260928 02:46:23.718669 1270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:23.721262 1284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:23.722558 1270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:23.722673 1270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a"
format_stamp: "Formatted at 2026-09-28 02:46:23 on dist-test-slave-r238"
server_key: "d571e573a08a6eb360c188d98c0e36ee"
server_key_iv: "2f95be5b22205cfe2571270f8a2afd13"
server_key_version: "encryptionkey@0"
I20260928 02:46:23.722767 1270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:23.735666 1270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:23.738794 1270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:23.738965 1270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:23.746222 1270 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:39063
I20260928 02:46:23.746515 1336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:39063 every 8 connection(s)
I20260928 02:46:23.746984 1270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:23.749993 1337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:23.750355 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1270
I20260928 02:46:23.750501 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:23.750804 29666 external_mini_cluster.cc:1428] Setting key ff5bcf598aa044994aeba2f3a6241cc4
I20260928 02:46:23.757377 1337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a: Bootstrap starting.
I20260928 02:46:23.760285 1337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a: Neither blocks nor log segments found. Creating new log.
Sep 28 02:46:23 dist-test-slave-r238 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:23.761129 1337 log.cc:824] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:23.763079 1337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a: No bootstrap required, opened a new log
I20260928 02:46:23.766237 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:23.752289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.28.248.190:39063)
0928 02:46:23.752764 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:23.752781 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:23.752817 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:46:23.753499 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:23.753518 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:23.753871 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:23.754167 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:23.755106 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:23.756099 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:23.757334 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:23.757985 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:23.758132 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:23.761094 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:23.761100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:23.761115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:23.761146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:23.763112 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:23.763638 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:23.763645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:23.763652 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:23.763786 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:23.764098 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:23.764103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:23.764105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:23.764541 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:23.764794 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:23.765108 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:46:23.765451 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":165,"threads_started":1}
I20260928 02:46:23.766816 1337 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39063 } }
I20260928 02:46:23.767067 1337 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:23.767140 1337 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a1d1ff555648b3a57bb42cd8a2cd5a, State: Initialized, Role: FOLLOWER
I20260928 02:46:23.767594 1337 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39063 } }
I20260928 02:46:23.767751 1337 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:23.767834 1337 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:23.767966 1337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:23.769083 1337 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39063 } }
I20260928 02:46:23.769433 1337 leader_election.cc:302] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a1d1ff555648b3a57bb42cd8a2cd5a; no voters:
I20260928 02:46:23.769752 1337 leader_election.cc:288] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:23.769912 1342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:23.770164 1342 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [term 1 LEADER]: Becoming Leader. State: Replica: d5a1d1ff555648b3a57bb42cd8a2cd5a, State: Running, Role: LEADER
I20260928 02:46:23.770509 1342 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39063 } }
I20260928 02:46:23.771095 1337 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:23.771697 1342 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a1d1ff555648b3a57bb42cd8a2cd5a. Latest consensus state: current_term: 1 leader_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39063 } } }
I20260928 02:46:23.771824 1342 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:23.772130 1344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1d1ff555648b3a57bb42cd8a2cd5a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39063 } } }
I20260928 02:46:23.772228 1344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:23.773217 1349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:23.775976 1349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:23.781498 1349 catalog_manager.cc:1347] Generated new cluster ID: 1f701d55c82a49399ff5ed75f7056d05
I20260928 02:46:23.781584 1349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:23.795415 1349 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:23.796466 1349 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:23.803089 1349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a: Generated new TSK 0
I20260928 02:46:23.803886 1349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:23.843201 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39063
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:23.957260 1363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:23.957540 1363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:23.957623 1363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:23.961791 1363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:23.961860 1363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:23.961987 1363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:23.967446 1363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:23.968760 1363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:23.969887 1363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:23.979800 1368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:23.979926 1369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:23.979800 1371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:23.980139 1363 server_base.cc:1034] running on GCE node
I20260928 02:46:23.980720 1363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:23.981423 1363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:23.982616 1363 hybrid_clock.cc:648] HybridClock initialized: now 1790563583982583 us; error 48 us; skew 500 ppm
Sep 28 02:46:23 dist-test-slave-r238 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:23.985669 1363 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:23.986469 1363 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:23.986572 1363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:23.986819 1363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:23.989048 1363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "e0f9cba2048d42278da2a69b990919d2"
format_stamp: "Formatted at 2026-09-28 02:46:23 on dist-test-slave-r238"
server_key: "92b4a7e134a5b16e8ba51bc5b1d59fcd"
server_key_iv: "385bef73670973880ca8ac39de4abe06"
server_key_version: "encryptionkey@0"
I20260928 02:46:23.989620 1363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "e0f9cba2048d42278da2a69b990919d2"
format_stamp: "Formatted at 2026-09-28 02:46:23 on dist-test-slave-r238"
server_key: "92b4a7e134a5b16e8ba51bc5b1d59fcd"
server_key_iv: "385bef73670973880ca8ac39de4abe06"
server_key_version: "encryptionkey@0"
I20260928 02:46:23.993670 1363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:46:23.996064 1377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:23.997357 1363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:23.997519 1363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "e0f9cba2048d42278da2a69b990919d2"
format_stamp: "Formatted at 2026-09-28 02:46:23 on dist-test-slave-r238"
server_key: "92b4a7e134a5b16e8ba51bc5b1d59fcd"
server_key_iv: "385bef73670973880ca8ac39de4abe06"
server_key_version: "encryptionkey@0"
I20260928 02:46:23.997643 1363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:24.018683 1363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:24.022256 1363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:24.022480 1363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:24.023120 1363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:24.023973 1363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:24.024047 1363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.024118 1363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:24.024169 1363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.033489 1363 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:36313
I20260928 02:46:24.033502 1490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:36313 every 8 connection(s)
I20260928 02:46:24.034117 1363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:24.040227 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1363
I20260928 02:46:24.040354 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:24.040688 29666 external_mini_cluster.cc:1428] Setting key b89e8dcb1e8f9b44a18f31ef9bffb5e7
Sep 28 02:46:24 dist-test-slave-r238 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563583, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:24.048313 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.035896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:37351 (local address 127.28.248.190:39063)
0928 02:46:24.036055 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.036059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.036964 (+ 905us) server_negotiation.cc:407] Connection header received
0928 02:46:24.038282 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.038285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.038335 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.038416 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:24.039636 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.040327 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.041685 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.041985 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.042077 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.044602 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:24.044638 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:24.044641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:24.044666 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:24.046194 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.046730 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.046735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.046736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.046819 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.047178 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.047182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.047183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.047376 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:24.047484 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.048004 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.048143 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:46:24.049242 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.036301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39063 (local address 127.28.248.129:37351)
0928 02:46:24.036819 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.036864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.038079 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.038577 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.038584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.038993 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:24.039503 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.039517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.040469 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.040475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.041563 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.041573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.042147 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.042151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.042332 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.042980 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:24.042995 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:24.044420 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:24.046320 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:24.046326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:24.046343 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:24.046622 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:24.046919 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:24.046922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:24.046924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:24.047074 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:24.047486 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:24.047495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:24.047754 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.047959 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.048298 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:46:24.050546 1491 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39063
I20260928 02:46:24.050843 1491 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:24.051311 1491 heartbeater.cc:511] Master 127.28.248.190:39063 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:24.053017 1301 ts_manager.cc:194] Registered new tserver with Master: e0f9cba2048d42278da2a69b990919d2 (127.28.248.129:36313)
I20260928 02:46:24.054169 1301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:37351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:24.068687 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39063
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:24.183784 1496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:24.184073 1496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:24.184120 1496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:24.188319 1496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:24.188403 1496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:24.188504 1496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:24.194036 1496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:24.195173 1496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:24.196234 1496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:24.205760 1501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:24.205811 1502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:24.205793 1504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:24.206156 1496 server_base.cc:1034] running on GCE node
I20260928 02:46:24.206573 1496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:24.207211 1496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:24.208398 1496 hybrid_clock.cc:648] HybridClock initialized: now 1790563584208387 us; error 33 us; skew 500 ppm
Sep 28 02:46:24 dist-test-slave-r238 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563584, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:24.211361 1496 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:24.212095 1496 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:24.212158 1496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:24.212330 1496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:24.214546 1496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "8e4ec83e19ec4724a6e5f0027adafebc"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "9f2e90a171f59c4c424a56a8286193fc"
server_key_iv: "77c7b4e414081a12e041f0bf430afcf4"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.215166 1496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "8e4ec83e19ec4724a6e5f0027adafebc"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "9f2e90a171f59c4c424a56a8286193fc"
server_key_iv: "77c7b4e414081a12e041f0bf430afcf4"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.219146 1496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:46:24.222014 1510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.223395 1496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:46:24.223690 1496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "8e4ec83e19ec4724a6e5f0027adafebc"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "9f2e90a171f59c4c424a56a8286193fc"
server_key_iv: "77c7b4e414081a12e041f0bf430afcf4"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.223891 1496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:24.248888 1496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:24.252849 1496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:24.253082 1496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:24.253756 1496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:24.254742 1496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:24.254803 1496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.254877 1496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:24.254932 1496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.266727 1496 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:38973
I20260928 02:46:24.266755 1623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:38973 every 8 connection(s)
I20260928 02:46:24.267359 1496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:24.275889 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1496
I20260928 02:46:24.276032 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:24.276314 29666 external_mini_cluster.cc:1428] Setting key b504ba8b5bdfb66668607c82024bb9d6
Sep 28 02:46:24 dist-test-slave-r238 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563584, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:24.284713 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.269220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:60821 (local address 127.28.248.190:39063)
0928 02:46:24.269377 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.269382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.270396 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 02:46:24.272562 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.272566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.272660 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.272750 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:24.274564 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.275089 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.277095 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.277389 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.277518 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.281008 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:24.281011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:24.281014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:24.281042 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:24.282604 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.283125 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.283130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.283132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.283214 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.283572 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.283577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.283579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.283771 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:24.283880 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.284392 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.284519 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:46:24.285621 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.269638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39063 (local address 127.28.248.130:60821)
0928 02:46:24.270181 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.270251 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.272252 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.273011 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.273024 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.273621 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:24.274400 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.274422 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.275221 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.275227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.276961 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.276975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.277566 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.277572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.277843 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.278635 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:24.278663 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:24.280833 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:24.282723 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:24.282729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:24.282748 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:24.283021 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:24.283312 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:24.283315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:24.283317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:24.283477 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:24.283878 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:24.283887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:24.284150 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.284343 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.284733 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260928 02:46:24.286782 1624 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39063
I20260928 02:46:24.287016 1624 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:24.287475 1624 heartbeater.cc:511] Master 127.28.248.190:39063 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:24.288518 1301 ts_manager.cc:194] Registered new tserver with Master: 8e4ec83e19ec4724a6e5f0027adafebc (127.28.248.130:38973)
I20260928 02:46:24.289227 1301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:60821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:24.308036 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39063
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:24.470131 1629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:24.470409 1629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:24.470456 1629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:24.474834 1629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:24.474911 1629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:24.474979 1629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:24.480693 1629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:24.481793 1629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:24.482795 1629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:24.492720 1634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:24.492730 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:24.492720 1637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:24.493132 1629 server_base.cc:1034] running on GCE node
I20260928 02:46:24.493595 1629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:24.494314 1629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:24.495486 1629 hybrid_clock.cc:648] HybridClock initialized: now 1790563584495468 us; error 29 us; skew 500 ppm
Sep 28 02:46:24 dist-test-slave-r238 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563584, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:24.498572 1629 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:24.499387 1629 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:24.499492 1629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:24.499749 1629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:24.502596 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "0140c1790c74450ab6bb7c6b95666424"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "ae1fa93ead8f0fb7ad93c1c87d07cc9c"
server_key_iv: "0be76ba2d277ade91fe20d26dae76db8"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.503207 1629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "0140c1790c74450ab6bb7c6b95666424"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "ae1fa93ead8f0fb7ad93c1c87d07cc9c"
server_key_iv: "0be76ba2d277ade91fe20d26dae76db8"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.507520 1629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:24.510255 1643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.511741 1629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 02:46:24.512045 1629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "0140c1790c74450ab6bb7c6b95666424"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "ae1fa93ead8f0fb7ad93c1c87d07cc9c"
server_key_iv: "0be76ba2d277ade91fe20d26dae76db8"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.512174 1629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:24.533864 1629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:24.537099 1629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:24.537330 1629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:24.537954 1629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:24.538823 1629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:24.538904 1629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.539001 1629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:24.539050 1629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.548434 1629 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:39049
I20260928 02:46:24.548454 1756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:39049 every 8 connection(s)
I20260928 02:46:24.549088 1629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:24.554714 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1629
I20260928 02:46:24.554863 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:24.555189 29666 external_mini_cluster.cc:1428] Setting key 8435831487a5259d87b9ebe2572de6b6
Sep 28 02:46:24 dist-test-slave-r238 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563584, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:24.563259 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.551003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:51861 (local address 127.28.248.190:39063)
0928 02:46:24.551134 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.551137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.552070 (+ 933us) server_negotiation.cc:407] Connection header received
0928 02:46:24.553644 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.553647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.553698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.553778 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:24.555186 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.555762 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.557228 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.557506 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.557587 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.559998 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:24.560001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:24.560003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:24.560025 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:24.561365 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.561931 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.561934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.561935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.562015 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.562314 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.562316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.562317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.562502 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:24.562575 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.563036 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.563138 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:46:24.564127 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.551375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39063 (local address 127.28.248.131:51861)
0928 02:46:24.551834 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.551879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.553431 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.553946 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.553953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.554334 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:24.555035 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.555057 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.555882 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.555885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.557069 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.557077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.557634 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.557637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.557777 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.558419 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:24.558435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:24.559854 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:24.561567 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:24.561572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:24.561584 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:24.561837 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:24.562102 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:24.562104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:24.562106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:24.562235 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:24.562589 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:24.562594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:24.562834 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.562993 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.563249 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260928 02:46:24.565340 1757 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39063
I20260928 02:46:24.565577 1757 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:24.565999 1757 heartbeater.cc:511] Master 127.28.248.190:39063 requested a full tablet report, sending...
I20260928 02:46:24.567016 1301 ts_manager.cc:194] Registered new tserver with Master: 0140c1790c74450ab6bb7c6b95666424 (127.28.248.131:39049)
I20260928 02:46:24.567642 1301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:51861
I20260928 02:46:24.569253 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:24.578629 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.571256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.28.248.190:39063)
0928 02:46:24.571372 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.571376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.571574 (+ 198us) server_negotiation.cc:407] Connection header received
0928 02:46:24.571674 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.571676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.571712 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.571818 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:24.572506 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.573114 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.573957 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.574160 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.574234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.575603 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:24.575605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:24.575607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:24.575624 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:24.577103 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.577518 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.577521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.577522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.577590 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:24.577861 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:24.577863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:24.577864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:24.578078 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:24.578176 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.578403 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.578509 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:46:24.581631 1301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:24.584551 1301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:24.598616 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.593598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.28.248.131:39049)
0928 02:46:24.594029 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.594045 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.594066 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:46:24.594136 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.594141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.594336 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.594742 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:24.594896 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.595681 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.597217 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.597917 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.598054 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.598189 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.598361 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.598452 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":124,"threads_started":1}
I20260928 02:46:24.598611 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.592983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:39049 (local address 127.0.0.1:53448)
0928 02:46:24.593520 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.593535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.593622 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.594485 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.594488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.594504 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:24.594602 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.594609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.595822 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.595825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.597081 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.597091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.598040 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.598044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.598147 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.598184 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.598329 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.598452 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260928 02:46:24.599064 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.592777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:36313 (local address 127.0.0.1:57128)
0928 02:46:24.593276 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.593325 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.593467 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.594248 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.594256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.594280 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:24.594421 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.594427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.595895 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.595899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.597010 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.597021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.598256 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.598260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.598355 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.598383 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.598474 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.598550 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":73,"threads_started":1}
I20260928 02:46:24.599076 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.593493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38973 (local address 127.0.0.1:40592)
0928 02:46:24.594121 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.594134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.594214 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.594884 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.594887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.594903 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:24.595004 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.595009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.596336 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.596340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.597514 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.597522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.598532 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.598535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.598623 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.598651 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.598864 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.598931 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":20,"thread_start_us":85,"threads_started":1}
I20260928 02:46:24.599251 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.592927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.28.248.129:36313)
0928 02:46:24.593354 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.593364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.593391 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:46:24.593691 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.593700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.593928 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.594073 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:24.594563 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.595764 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.597140 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.597927 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.598052 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.598190 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.598682 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.599018 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260928 02:46:24.599844 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.593914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.28.248.130:38973)
0928 02:46:24.594457 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.594470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.594493 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:46:24.594564 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.594569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.594737 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.594885 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:24.595134 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.596201 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.597636 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.598406 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.599254 (+ 848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.599542 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.599598 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.599699 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":86,"threads_started":1}
I20260928 02:46:24.602072 1425 tablet_service.cc:1467] Processing CreateTablet for tablet 7c154034e56348798cc90ebbbd699234 (DEFAULT_TABLE table=test-table [id=1001b4d21ff44b768cbe82410089d8b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:24.602102 1691 tablet_service.cc:1467] Processing CreateTablet for tablet 7c154034e56348798cc90ebbbd699234 (DEFAULT_TABLE table=test-table [id=1001b4d21ff44b768cbe82410089d8b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:24.602569 1558 tablet_service.cc:1467] Processing CreateTablet for tablet 7c154034e56348798cc90ebbbd699234 (DEFAULT_TABLE table=test-table [id=1001b4d21ff44b768cbe82410089d8b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:24.603157 1425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c154034e56348798cc90ebbbd699234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:24.603188 1691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c154034e56348798cc90ebbbd699234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:24.603482 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c154034e56348798cc90ebbbd699234. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:24.610289 1774 tablet_bootstrap.cc:492] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: Bootstrap starting.
I20260928 02:46:24.610810 1775 tablet_bootstrap.cc:492] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Bootstrap starting.
I20260928 02:46:24.610837 1776 tablet_bootstrap.cc:492] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Bootstrap starting.
I20260928 02:46:24.613122 1775 tablet_bootstrap.cc:654] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:24.613142 1776 tablet_bootstrap.cc:654] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:24.613142 1774 tablet_bootstrap.cc:654] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:24.614122 1774 log.cc:824] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:24.614343 1775 log.cc:824] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:24.614370 1776 log.cc:824] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:24.616367 1776 tablet_bootstrap.cc:492] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: No bootstrap required, opened a new log
I20260928 02:46:24.616367 1774 tablet_bootstrap.cc:492] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: No bootstrap required, opened a new log
I20260928 02:46:24.616575 1776 ts_tablet_manager.cc:1397] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:46:24.616575 1774 ts_tablet_manager.cc:1397] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:46:24.616410 1775 tablet_bootstrap.cc:492] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: No bootstrap required, opened a new log
I20260928 02:46:24.616820 1775 ts_tablet_manager.cc:1397] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:46:24.620105 1775 raft_consensus.cc:348] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.620395 1775 raft_consensus.cc:374] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:24.620461 1775 raft_consensus.cc:729] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0140c1790c74450ab6bb7c6b95666424, State: Initialized, Role: FOLLOWER
I20260928 02:46:24.620633 1776 raft_consensus.cc:348] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.620918 1776 raft_consensus.cc:374] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:24.620973 1776 raft_consensus.cc:729] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4ec83e19ec4724a6e5f0027adafebc, State: Initialized, Role: FOLLOWER
I20260928 02:46:24.620803 1774 raft_consensus.cc:348] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.621039 1774 raft_consensus.cc:374] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:24.621093 1774 raft_consensus.cc:729] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f9cba2048d42278da2a69b990919d2, State: Initialized, Role: FOLLOWER
I20260928 02:46:24.621062 1775 consensus_queue.cc:260] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.621421 1776 consensus_queue.cc:260] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.621452 1774 consensus_queue.cc:260] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.621897 1757 heartbeater.cc:503] Master 127.28.248.190:39063 was elected leader, sending a full tablet report...
I20260928 02:46:24.622285 1775 ts_tablet_manager.cc:1428] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:46:24.622359 1776 ts_tablet_manager.cc:1428] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:46:24.622696 1624 heartbeater.cc:503] Master 127.28.248.190:39063 was elected leader, sending a full tablet report...
I20260928 02:46:24.623301 1491 heartbeater.cc:503] Master 127.28.248.190:39063 was elected leader, sending a full tablet report...
I20260928 02:46:24.623638 1774 ts_tablet_manager.cc:1428] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:46:24.668413 1780 raft_consensus.cc:482] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:24.668720 1780 raft_consensus.cc:504] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.670209 1780 leader_election.cc:288] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0f9cba2048d42278da2a69b990919d2 (127.28.248.129:36313), 8e4ec83e19ec4724a6e5f0027adafebc (127.28.248.130:38973)
I20260928 02:46:24.674602 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.670590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:34057 (local address 127.28.248.129:36313)
0928 02:46:24.670721 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.670726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.670933 (+ 207us) server_negotiation.cc:407] Connection header received
0928 02:46:24.671096 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.671102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.671159 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.671232 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:24.671761 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.672381 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.673570 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.673985 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.674118 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.674171 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.674383 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.674452 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:24.674618 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.670382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:36313 (local address 127.28.248.131:34057)
0928 02:46:24.670833 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.670850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.670941 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.671441 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.671443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.671457 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:24.671638 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.671645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.672492 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.672498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.673440 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.673449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.674108 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.674112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.674226 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.674278 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.674367 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.674452 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":9}
I20260928 02:46:24.674605 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.670733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:33113 (local address 127.28.248.130:38973)
0928 02:46:24.670865 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.670869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.671149 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:46:24.671243 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.671245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.671284 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.671343 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:24.671709 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.672329 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.673578 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.673985 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.674119 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.674172 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.674413 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.674481 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:46:24.674643 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.670498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38973 (local address 127.28.248.131:33113)
0928 02:46:24.671039 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.671055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:24.671160 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:24.671390 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:24.671392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:24.671406 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:24.671578 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.671588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.672447 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.672451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.673456 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:24.673466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.674110 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.674114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:24.674291 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.674310 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:46:24.674396 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:46:24.674478 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":161,"thread_start_us":122,"threads_started":1}
I20260928 02:46:24.675321 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154034e56348798cc90ebbbd699234" candidate_uuid: "0140c1790c74450ab6bb7c6b95666424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f9cba2048d42278da2a69b990919d2" is_pre_election: true
I20260928 02:46:24.675698 1445 raft_consensus.cc:2454] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0140c1790c74450ab6bb7c6b95666424 in term 0.
I20260928 02:46:24.675972 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154034e56348798cc90ebbbd699234" candidate_uuid: "0140c1790c74450ab6bb7c6b95666424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" is_pre_election: true
I20260928 02:46:24.676245 1645 leader_election.cc:302] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140c1790c74450ab6bb7c6b95666424, e0f9cba2048d42278da2a69b990919d2; no voters:
I20260928 02:46:24.676270 1578 raft_consensus.cc:2454] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0140c1790c74450ab6bb7c6b95666424 in term 0.
I20260928 02:46:24.676541 1780 raft_consensus.cc:2781] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:24.676674 1780 raft_consensus.cc:482] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:24.676736 1780 raft_consensus.cc:3037] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:24.677858 1780 raft_consensus.cc:504] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.678224 1780 leader_election.cc:288] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [CANDIDATE]: Term 1 election: Requested vote from peers e0f9cba2048d42278da2a69b990919d2 (127.28.248.129:36313), 8e4ec83e19ec4724a6e5f0027adafebc (127.28.248.130:38973)
I20260928 02:46:24.678603 1578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154034e56348798cc90ebbbd699234" candidate_uuid: "0140c1790c74450ab6bb7c6b95666424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4ec83e19ec4724a6e5f0027adafebc"
I20260928 02:46:24.678707 1578 raft_consensus.cc:3037] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:24.679695 1578 raft_consensus.cc:2454] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0140c1790c74450ab6bb7c6b95666424 in term 1.
I20260928 02:46:24.680006 1646 leader_election.cc:302] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140c1790c74450ab6bb7c6b95666424, 8e4ec83e19ec4724a6e5f0027adafebc; no voters:
I20260928 02:46:24.679991 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154034e56348798cc90ebbbd699234" candidate_uuid: "0140c1790c74450ab6bb7c6b95666424" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f9cba2048d42278da2a69b990919d2"
I20260928 02:46:24.680120 1445 raft_consensus.cc:3037] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:24.680230 1780 raft_consensus.cc:2781] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:24.680464 1780 raft_consensus.cc:686] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 1 LEADER]: Becoming Leader. State: Replica: 0140c1790c74450ab6bb7c6b95666424, State: Running, Role: LEADER
I20260928 02:46:24.680846 1780 consensus_queue.cc:237] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } }
I20260928 02:46:24.681383 1445 raft_consensus.cc:2454] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0140c1790c74450ab6bb7c6b95666424 in term 1.
I20260928 02:46:24.684057 1301 catalog_manager.cc:5495] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0140c1790c74450ab6bb7c6b95666424 (127.28.248.131). New cstate: current_term: 1 leader_uuid: "0140c1790c74450ab6bb7c6b95666424" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0140c1790c74450ab6bb7c6b95666424" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:24.697680 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.694390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.28.248.131:39049)
0928 02:46:24.694516 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.694521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.694543 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:46:24.694633 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.694635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.694688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.694771 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:24.695026 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.695522 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.696318 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:24.696522 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:24.696595 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:24.696906 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:24.697346 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:24.697446 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.697500 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.697552 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:24.703903 1578 raft_consensus.cc:1261] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 1 FOLLOWER]: Refusing update from remote peer 0140c1790c74450ab6bb7c6b95666424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:24.703903 1445 raft_consensus.cc:1261] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2 [term 1 FOLLOWER]: Refusing update from remote peer 0140c1790c74450ab6bb7c6b95666424: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:24.704700 1784 consensus_queue.cc:1035] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4ec83e19ec4724a6e5f0027adafebc" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:24.704903 1780 consensus_queue.cc:1035] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f9cba2048d42278da2a69b990919d2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:24.713878 1788 mvcc.cc:205] Tried to move back new op lower bound from 7334148442940055552 to 7334148442855641088. Current Snapshot: MvccSnapshot[applied={T|T < 7334148442940055552}]
I20260928 02:46:24.721377 1301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:24.721578 1301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:24.724201 1301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a1d1ff555648b3a57bb42cd8a2cd5a: Sending DeleteTablet for 3 replicas of tablet 7c154034e56348798cc90ebbbd699234
I20260928 02:46:24.725106 1425 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c154034e56348798cc90ebbbd699234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:24 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:57128
I20260928 02:46:24.725147 1558 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c154034e56348798cc90ebbbd699234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:24 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:40592
I20260928 02:46:24.725360 1691 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c154034e56348798cc90ebbbd699234 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:24 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:53448
I20260928 02:46:24.725674 1797 tablet_replica.cc:330] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: stopping tablet replica
I20260928 02:46:24.725674 1798 tablet_replica.cc:330] T 7c154034e56348798cc90ebbbd699234 P e0f9cba2048d42278da2a69b990919d2: stopping tablet replica
I20260928 02:46:24.726052 1797 raft_consensus.cc:2229] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:24.726154 1799 tablet_replica.cc:330] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: stopping tablet replica
I20260928 02:46:24.726272 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1363
I20260928 02:46:24.726452 1799 raft_consensus.cc:2229] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:24.726608 1797 raft_consensus.cc:2258] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:24.726831 1799 raft_consensus.cc:2258] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:24.728916 1797 ts_tablet_manager.cc:1905] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:24.732687 1797 ts_tablet_manager.cc:1918] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:24.732872 1797 log.cc:1196] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/7c154034e56348798cc90ebbbd699234
I20260928 02:46:24.733304 1797 ts_tablet_manager.cc:1939] T 7c154034e56348798cc90ebbbd699234 P 8e4ec83e19ec4724a6e5f0027adafebc: Deleting consensus metadata
I20260928 02:46:24.734169 1799 ts_tablet_manager.cc:1905] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:24.734862 1287 catalog_manager.cc:4841] TS 8e4ec83e19ec4724a6e5f0027adafebc (127.28.248.130:38973): tablet 7c154034e56348798cc90ebbbd699234 (table test-table [id=1001b4d21ff44b768cbe82410089d8b9]) successfully deleted
W20260928 02:46:24.736115 1286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:36313 (error 108)
I20260928 02:46:24.736804 1799 ts_tablet_manager.cc:1918] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:24.736907 1799 log.cc:1196] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/7c154034e56348798cc90ebbbd699234
I20260928 02:46:24.737248 1799 ts_tablet_manager.cc:1939] T 7c154034e56348798cc90ebbbd699234 P 0140c1790c74450ab6bb7c6b95666424: Deleting consensus metadata
I20260928 02:46:24.737316 1768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.737013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:36313 (local address 127.0.0.1:57136)
0928 02:46:24.737135 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.737210 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:46:24.737416 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1496
W20260928 02:46:24.738052 1286 catalog_manager.cc:4584] TS e0f9cba2048d42278da2a69b990919d2 (127.28.248.129:36313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c154034e56348798cc90ebbbd699234: Network error: Client connection negotiation failed: client connection to 127.28.248.129:36313: connect: Connection refused (error 111)
I20260928 02:46:24.739028 1287 catalog_manager.cc:4841] TS 0140c1790c74450ab6bb7c6b95666424 (127.28.248.131:39049): tablet 7c154034e56348798cc90ebbbd699234 (table test-table [id=1001b4d21ff44b768cbe82410089d8b9]) successfully deleted
I20260928 02:46:24.744674 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1629
I20260928 02:46:24.749935 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1270
2026-09-28T02:46:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2352 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:24Z Disabled control of system clock
I20260928 02:46:24.779484 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:37147
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:37147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:46:24.893288 1804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:24.893589 1804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:24.893626 1804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:24.897822 1804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:24.897897 1804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:24.897912 1804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:24.897930 1804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:24.897959 1804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:24.902778 1804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:37147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:37147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:24.903872 1804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:24.905354 1804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:24.914220 1810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:24.914225 1809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:24.914345 1804 server_base.cc:1034] running on GCE node
W20260928 02:46:24.914225 1812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:24.915215 1804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:24.916426 1804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:24.917593 1804 hybrid_clock.cc:648] HybridClock initialized: now 1790563584917574 us; error 41 us; skew 500 ppm
I20260928 02:46:24.919519 1804 webserver.cc:460] Webserver started at http://127.28.248.190:36861/ using document root <none> and password file <none>
I20260928 02:46:24.920127 1804 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:24.920192 1804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:24.920431 1804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:24.922606 1804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "744cc1c783d64ec68b0aa443d366878a"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "fe9b236aca07645c1e8ba8f877bff6f3"
server_key_iv: "ece4119705bbd994b2f6c5801f20d7de"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.923206 1804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "744cc1c783d64ec68b0aa443d366878a"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "fe9b236aca07645c1e8ba8f877bff6f3"
server_key_iv: "ece4119705bbd994b2f6c5801f20d7de"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.927232 1804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:46:24.929821 1818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:24.931133 1804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:24.931277 1804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "744cc1c783d64ec68b0aa443d366878a"
format_stamp: "Formatted at 2026-09-28 02:46:24 on dist-test-slave-r238"
server_key: "fe9b236aca07645c1e8ba8f877bff6f3"
server_key_iv: "ece4119705bbd994b2f6c5801f20d7de"
server_key_version: "encryptionkey@0"
I20260928 02:46:24.931399 1804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:24.936182 1804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:24.936944 1804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:24.937150 1804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:24.944324 1804 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:37147
I20260928 02:46:24.944348 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:37147 every 8 connection(s)
I20260928 02:46:24.945402 1804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:24.948837 1871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:24.954603 1871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a: Bootstrap starting.
I20260928 02:46:24.955550 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1804
I20260928 02:46:24.955672 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:24.955999 29666 external_mini_cluster.cc:1428] Setting key d4b10940e02d4e7634a182d25d95dcd9
I20260928 02:46:24.957700 1871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:24.958477 1871 log.cc:824] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:24.960949 1871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a: No bootstrap required, opened a new log
I20260928 02:46:24.961730 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.957218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.28.248.190:37147)
0928 02:46:24.957885 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:24.957905 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:24.957945 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:46:24.958652 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:24.958671 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:24.959123 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:24.959481 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:24.960310 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:24.960315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:24.960327 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:24.960336 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:24.960406 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:24.960422 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:24.960510 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:24.960699 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:46:24.960963 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260928 02:46:24.964927 1871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744cc1c783d64ec68b0aa443d366878a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37147 } }
I20260928 02:46:24.965153 1871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:24.965230 1871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744cc1c783d64ec68b0aa443d366878a, State: Initialized, Role: FOLLOWER
I20260928 02:46:24.965670 1871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744cc1c783d64ec68b0aa443d366878a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37147 } }
I20260928 02:46:24.965816 1871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:24.965883 1871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:24.966012 1871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:24.966988 1871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744cc1c783d64ec68b0aa443d366878a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37147 } }
I20260928 02:46:24.967348 1871 leader_election.cc:302] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 744cc1c783d64ec68b0aa443d366878a; no voters:
I20260928 02:46:24.967710 1871 leader_election.cc:288] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:24.967787 1876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:24.967974 1876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [term 1 LEADER]: Becoming Leader. State: Replica: 744cc1c783d64ec68b0aa443d366878a, State: Running, Role: LEADER
I20260928 02:46:24.968297 1876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744cc1c783d64ec68b0aa443d366878a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37147 } }
I20260928 02:46:24.969106 1871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:24.969976 1878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 744cc1c783d64ec68b0aa443d366878a. Latest consensus state: current_term: 1 leader_uuid: "744cc1c783d64ec68b0aa443d366878a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744cc1c783d64ec68b0aa443d366878a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37147 } } }
I20260928 02:46:24.970110 1878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:24.970719 1884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:24.971375 1877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "744cc1c783d64ec68b0aa443d366878a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744cc1c783d64ec68b0aa443d366878a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37147 } } }
I20260928 02:46:24.971477 1877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:24.974265 1884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:24.979911 1884 catalog_manager.cc:1347] Generated new cluster ID: 23df595483564ab29b5c27daa2860152
I20260928 02:46:24.979990 1884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:25.023964 1884 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:25.024461 1884 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:25.031987 1884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a: Generated new TSK 0
I20260928 02:46:25.032774 1884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:25.045258 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37147
--builtin_ntp_servers=127.28.248.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:46:25.158706 1895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:25.158993 1895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:25.159076 1895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:25.163208 1895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:25.163277 1895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:25.163381 1895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:25.168313 1895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:25.169579 1895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:25.171235 1895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:46:25.180893 1895 server_base.cc:1034] running on GCE node
W20260928 02:46:25.180857 1900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:25.180857 1903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:25.180897 1901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:25.181523 1895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:25.182153 1895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:25.183351 1895 hybrid_clock.cc:648] HybridClock initialized: now 1790563585183337 us; error 25 us; skew 500 ppm
I20260928 02:46:25.185256 1895 webserver.cc:460] Webserver started at http://127.28.248.129:40647/ using document root <none> and password file <none>
I20260928 02:46:25.185844 1895 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:25.185940 1895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:25.186169 1895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:25.188236 1895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "49dccc1c4dd5432a8a74255a98fa1df9"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "0bd2715e0e3b95a2b163ba67f0340f20"
server_key_iv: "2a10cc06f047a62141d946d57bfbb264"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.188867 1895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "49dccc1c4dd5432a8a74255a98fa1df9"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "0bd2715e0e3b95a2b163ba67f0340f20"
server_key_iv: "2a10cc06f047a62141d946d57bfbb264"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.192469 1895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:25.194948 1909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.196161 1895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:25.196291 1895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "49dccc1c4dd5432a8a74255a98fa1df9"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "0bd2715e0e3b95a2b163ba67f0340f20"
server_key_iv: "2a10cc06f047a62141d946d57bfbb264"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.196416 1895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:25.200990 1895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:25.201612 1895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:25.201795 1895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:25.202332 1895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:25.203218 1895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:25.203287 1895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.203356 1895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:25.203409 1895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.212662 1895 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:46343
I20260928 02:46:25.212680 2022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:46343 every 8 connection(s)
I20260928 02:46:25.213578 1895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:25.219230 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.215325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:48185 (local address 127.28.248.190:37147)
0928 02:46:25.215464 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.215468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.216502 (+ 1034us) server_negotiation.cc:407] Connection header received
0928 02:46:25.217766 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.217768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.217804 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.217875 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.218855 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.218858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.218860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.218870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.218911 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.218919 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.218992 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.219089 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.219127 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:25.220250 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.215732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37147 (local address 127.28.248.129:48185)
0928 02:46:25.216363 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.216402 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.217579 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.218012 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.218020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.218105 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.218610 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.218612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.218651 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.218716 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.218977 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.218981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.218984 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.219104 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.219318 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260928 02:46:25.221451 2023 heartbeater.cc:347] Connected to a master server at 127.28.248.190:37147
I20260928 02:46:25.221664 2023 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:25.221940 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 1895
I20260928 02:46:25.222069 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:25.222143 2023 heartbeater.cc:511] Master 127.28.248.190:37147 requested a full tablet report, sending...
I20260928 02:46:25.222381 29666 external_mini_cluster.cc:1428] Setting key 21f85b742411bf889b49904dda1e250a
I20260928 02:46:25.223688 1835 ts_manager.cc:194] Registered new tserver with Master: 49dccc1c4dd5432a8a74255a98fa1df9 (127.28.248.129:46343)
I20260928 02:46:25.224370 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37147
--builtin_ntp_servers=127.28.248.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:46:25.335554 2026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:25.335937 2026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:25.336037 2026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:25.340233 2026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:25.340291 2026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:25.340394 2026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:25.345283 2026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:25.346374 2026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:25.347764 2026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:25.357573 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:25.357558 2032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:25.357618 2026 server_base.cc:1034] running on GCE node
W20260928 02:46:25.357560 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:25.358274 2026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:25.358932 2026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:25.360101 2026 hybrid_clock.cc:648] HybridClock initialized: now 1790563585360082 us; error 30 us; skew 500 ppm
I20260928 02:46:25.362020 2026 webserver.cc:460] Webserver started at http://127.28.248.130:36583/ using document root <none> and password file <none>
I20260928 02:46:25.362614 2026 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:25.362679 2026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:25.362902 2026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:25.365070 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "2597748504804b3c872c707804f5ea1b"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "8851a4cbda9d21a9d7112171795dd253"
server_key_iv: "d15504a930220e62a3cab747bb6e56e1"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.365643 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "2597748504804b3c872c707804f5ea1b"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "8851a4cbda9d21a9d7112171795dd253"
server_key_iv: "d15504a930220e62a3cab747bb6e56e1"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.369541 2026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:46:25.372720 2040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.374063 2026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:46:25.374195 2026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "2597748504804b3c872c707804f5ea1b"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "8851a4cbda9d21a9d7112171795dd253"
server_key_iv: "d15504a930220e62a3cab747bb6e56e1"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.374323 2026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:25.380158 2026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:25.380982 2026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:25.381179 2026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:25.381780 2026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:25.382755 2026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:25.382804 2026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.382845 2026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:25.382864 2026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.392201 2026 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:36319
I20260928 02:46:25.392221 2153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:36319 every 8 connection(s)
I20260928 02:46:25.393273 2026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:25.399734 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.395263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:35559 (local address 127.28.248.190:37147)
0928 02:46:25.395471 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.395477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.396637 (+ 1160us) server_negotiation.cc:407] Connection header received
0928 02:46:25.397977 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.397980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.398017 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.398102 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.399212 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.399216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.399218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.399230 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.399252 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.399263 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.399350 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.399576 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.399619 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 02:46:25.400728 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2026
I20260928 02:46:25.400741 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.395772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37147 (local address 127.28.248.130:35559)
0928 02:46:25.396451 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.396493 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.397783 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.398268 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.398274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.398366 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.398903 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.398905 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.398946 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.399016 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.399373 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.399381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.399386 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.399543 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.399739 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260928 02:46:25.400862 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:25.401151 29666 external_mini_cluster.cc:1428] Setting key a27b8ee1f0b70b83fd3b0b5b5377f879
I20260928 02:46:25.402043 2154 heartbeater.cc:347] Connected to a master server at 127.28.248.190:37147
I20260928 02:46:25.402297 2154 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:25.402921 2154 heartbeater.cc:511] Master 127.28.248.190:37147 requested a full tablet report, sending...
I20260928 02:46:25.403317 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37147
--builtin_ntp_servers=127.28.248.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:46:25.403988 1835 ts_manager.cc:194] Registered new tserver with Master: 2597748504804b3c872c707804f5ea1b (127.28.248.130:36319)
W20260928 02:46:25.515959 2157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:25.516271 2157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:25.516350 2157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:25.520403 2157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:25.520485 2157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:25.520581 2157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:25.525439 2157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:25.526615 2157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:25.528013 2157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:25.537835 2165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:25.537916 2162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:25.537901 2163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:25.538070 2157 server_base.cc:1034] running on GCE node
I20260928 02:46:25.538506 2157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:25.539250 2157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:25.540432 2157 hybrid_clock.cc:648] HybridClock initialized: now 1790563585540430 us; error 41 us; skew 500 ppm
I20260928 02:46:25.542263 2157 webserver.cc:460] Webserver started at http://127.28.248.131:41775/ using document root <none> and password file <none>
I20260928 02:46:25.542842 2157 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:25.542933 2157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:25.543222 2157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:25.545418 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "4eb1ff2a629e48298e14b6e824a66fa7"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "312ccb63d1d59d3369e3783a6c7789f6"
server_key_iv: "db2ba4f55e69306ed0cd910babe762e8"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.545991 2157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "4eb1ff2a629e48298e14b6e824a66fa7"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "312ccb63d1d59d3369e3783a6c7789f6"
server_key_iv: "db2ba4f55e69306ed0cd910babe762e8"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.549626 2157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:25.552104 2171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.553459 2157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:25.553587 2157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "4eb1ff2a629e48298e14b6e824a66fa7"
format_stamp: "Formatted at 2026-09-28 02:46:25 on dist-test-slave-r238"
server_key: "312ccb63d1d59d3369e3783a6c7789f6"
server_key_iv: "db2ba4f55e69306ed0cd910babe762e8"
server_key_version: "encryptionkey@0"
I20260928 02:46:25.553700 2157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:25.558203 2157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:25.558794 2157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:25.558984 2157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:25.559558 2157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:25.560428 2157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:25.560500 2157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.560562 2157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:25.560614 2157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:25.569324 2157 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:38233
I20260928 02:46:25.569353 2284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:38233 every 8 connection(s)
I20260928 02:46:25.570380 2157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:25.576506 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.572284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:32841 (local address 127.28.248.190:37147)
0928 02:46:25.572424 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.572428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.573524 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 02:46:25.574818 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.574821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.574863 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.574960 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.576040 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.576044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.576046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.576058 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.576079 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.576088 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.576188 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.576326 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.576379 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:46:25.577334 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.572757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37147 (local address 127.28.248.131:32841)
0928 02:46:25.573387 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.573426 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.574609 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.575114 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.575121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.575205 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.575789 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.575791 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.575829 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.575898 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.576182 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.576189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.576197 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.576366 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.576588 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260928 02:46:25.578538 2285 heartbeater.cc:347] Connected to a master server at 127.28.248.190:37147
I20260928 02:46:25.578739 2285 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:25.579131 2285 heartbeater.cc:511] Master 127.28.248.190:37147 requested a full tablet report, sending...
I20260928 02:46:25.579866 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2157
I20260928 02:46:25.579993 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:25.580159 1835 ts_manager.cc:194] Registered new tserver with Master: 4eb1ff2a629e48298e14b6e824a66fa7 (127.28.248.131:38233)
I20260928 02:46:25.580296 29666 external_mini_cluster.cc:1428] Setting key 1b06e149fbffb71943c95210465da3dc
I20260928 02:46:25.582140 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:25.585642 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.583941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.28.248.190:37147)
0928 02:46:25.584100 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.584104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.584220 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:46:25.584358 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.584361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.584393 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.584485 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.585254 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.585258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.585260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.585280 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.585298 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.585307 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.585400 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.585482 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.585524 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:25.588284 1835 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:25.591439 1835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:25.603652 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.600454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:46343 (local address 127.0.0.1:35500)
0928 02:46:25.601290 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.601305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.601376 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.601890 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.601903 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.601980 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.602615 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.602617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.602640 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.602709 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.603299 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.603308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.603314 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.603431 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.603500 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":106,"threads_started":1}
I20260928 02:46:25.603716 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.600902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.28.248.129:46343)
0928 02:46:25.601367 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.601379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.601429 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:46:25.601494 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.601504 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.601729 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.601899 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.603127 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.603132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.603141 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.603149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.603180 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.603195 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.603321 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.603482 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.603566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":105,"threads_started":1}
I20260928 02:46:25.603865 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.601095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:36319 (local address 127.0.0.1:54630)
0928 02:46:25.601883 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.601898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.601984 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.602611 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.602614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.602628 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.603035 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.603037 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.603048 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.603075 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.603491 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.603494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.603498 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.603552 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.603583 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":95,"threads_started":1}
I20260928 02:46:25.604416 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.601739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.28.248.130:36319)
0928 02:46:25.602257 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.602276 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.602306 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:46:25.602350 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.602354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.602485 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.602627 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.603300 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.603304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.603313 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.603321 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.603348 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.603363 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.604100 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.604156 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.604239 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":114,"threads_started":1}
I20260928 02:46:25.606292 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.600375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:38233 (local address 127.0.0.1:41332)
0928 02:46:25.601108 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.601161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.601354 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.603961 (+ 2607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.603964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.603978 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.604561 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.604563 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.604573 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.604598 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.606051 (+ 1453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.606055 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.606058 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.606144 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.606178 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":77,"threads_started":1}
I20260928 02:46:25.606379 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.600487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.28.248.131:38233)
0928 02:46:25.603532 (+ 3045us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.603543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.603566 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:46:25.603615 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.603619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.603788 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.605268 (+ 1480us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.605862 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.605867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.605878 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.605886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.605922 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.605945 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.606052 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.606151 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.606215 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2941,"thread_start_us":164,"threads_started":1}
I20260928 02:46:25.607260 1957 tablet_service.cc:1467] Processing CreateTablet for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 (DEFAULT_TABLE table=test-table [id=180fa49d0ccf452d87e5bb180c335c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:25.607420 2088 tablet_service.cc:1467] Processing CreateTablet for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 (DEFAULT_TABLE table=test-table [id=180fa49d0ccf452d87e5bb180c335c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:25.608461 1957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:25.608470 2088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:25.610610 2219 tablet_service.cc:1467] Processing CreateTablet for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 (DEFAULT_TABLE table=test-table [id=180fa49d0ccf452d87e5bb180c335c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:25.611625 2219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:25.614753 2302 tablet_bootstrap.cc:492] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Bootstrap starting.
I20260928 02:46:25.616283 2303 tablet_bootstrap.cc:492] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: Bootstrap starting.
I20260928 02:46:25.617213 2302 tablet_bootstrap.cc:654] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:25.617986 2302 log.cc:824] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:25.618595 2303 tablet_bootstrap.cc:654] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:25.619423 2303 log.cc:824] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:25.620025 2302 tablet_bootstrap.cc:492] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: No bootstrap required, opened a new log
I20260928 02:46:25.620234 2302 ts_tablet_manager.cc:1397] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:46:25.618748 2305 tablet_bootstrap.cc:492] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Bootstrap starting.
I20260928 02:46:25.621205 2303 tablet_bootstrap.cc:492] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: No bootstrap required, opened a new log
I20260928 02:46:25.621407 2303 ts_tablet_manager.cc:1397] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:46:25.622884 2305 tablet_bootstrap.cc:654] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:25.623797 2305 log.cc:824] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:25.624903 2302 raft_consensus.cc:348] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.625180 2302 raft_consensus.cc:374] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:25.625242 2302 raft_consensus.cc:729] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2597748504804b3c872c707804f5ea1b, State: Initialized, Role: FOLLOWER
I20260928 02:46:25.625379 2303 raft_consensus.cc:348] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.625670 2303 raft_consensus.cc:374] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:25.625727 2303 raft_consensus.cc:729] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49dccc1c4dd5432a8a74255a98fa1df9, State: Initialized, Role: FOLLOWER
I20260928 02:46:25.625712 2302 consensus_queue.cc:260] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.625907 2305 tablet_bootstrap.cc:492] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: No bootstrap required, opened a new log
I20260928 02:46:25.626102 2305 ts_tablet_manager.cc:1397] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260928 02:46:25.626257 2303 consensus_queue.cc:260] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.626441 2154 heartbeater.cc:503] Master 127.28.248.190:37147 was elected leader, sending a full tablet report...
I20260928 02:46:25.627089 2303 ts_tablet_manager.cc:1428] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:46:25.627453 2023 heartbeater.cc:503] Master 127.28.248.190:37147 was elected leader, sending a full tablet report...
I20260928 02:46:25.628012 2302 ts_tablet_manager.cc:1428] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:46:25.630887 2305 raft_consensus.cc:348] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.631129 2305 raft_consensus.cc:374] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:25.631207 2305 raft_consensus.cc:729] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb1ff2a629e48298e14b6e824a66fa7, State: Initialized, Role: FOLLOWER
I20260928 02:46:25.631642 2305 consensus_queue.cc:260] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.632287 2285 heartbeater.cc:503] Master 127.28.248.190:37147 was elected leader, sending a full tablet report...
I20260928 02:46:25.632527 2305 ts_tablet_manager.cc:1428] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 02:46:25.644299 2155 tablet.cc:2369] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:46:25.714776 2024 tablet.cc:2369] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:25.768704 2309 raft_consensus.cc:482] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:25.768936 2309 raft_consensus.cc:504] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.770334 2309 leader_election.cc:288] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb1ff2a629e48298e14b6e824a66fa7 (127.28.248.131:38233), 2597748504804b3c872c707804f5ea1b (127.28.248.130:36319)
I20260928 02:46:25.772732 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.770695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:56357 (local address 127.28.248.131:38233)
0928 02:46:25.770835 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.770840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.771141 (+ 301us) server_negotiation.cc:407] Connection header received
0928 02:46:25.771266 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.771269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.771312 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.771406 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.772288 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.772292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.772295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.772308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.772331 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.772341 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.772439 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.772519 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.772564 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:46:25.772732 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.770803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:38053 (local address 127.28.248.130:36319)
0928 02:46:25.770955 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.770960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.771131 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:46:25.771269 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.771272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.771312 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.771405 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.772288 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.772292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.772295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.772308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.772331 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.772341 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.772439 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.772519 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.772564 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:46:25.772789 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.770613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:38233 (local address 127.28.248.129:56357)
0928 02:46:25.771007 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.771023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.771158 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.771442 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.771445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.771463 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.772125 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.772127 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.772134 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.772150 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.772437 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.772440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.772443 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.772527 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.772597 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":21,"thread_start_us":107,"threads_started":1}
I20260928 02:46:25.772797 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.770700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:36319 (local address 127.28.248.129:38053)
0928 02:46:25.771051 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.771065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:25.771174 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:25.771442 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:25.771445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:25.771463 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:25.772083 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:25.772086 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:25.772101 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:25.772135 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:25.772437 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:25.772440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:25.772443 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:25.772527 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:46:25.772584 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":63,"threads_started":1}
I20260928 02:46:25.773588 2239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ae9bf4e0dc45b891c16ea25a2a2e04" candidate_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" is_pre_election: true
I20260928 02:46:25.773927 2239 raft_consensus.cc:2454] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49dccc1c4dd5432a8a74255a98fa1df9 in term 0.
I20260928 02:46:25.774406 2108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ae9bf4e0dc45b891c16ea25a2a2e04" candidate_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2597748504804b3c872c707804f5ea1b" is_pre_election: true
I20260928 02:46:25.774503 1910 leader_election.cc:302] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49dccc1c4dd5432a8a74255a98fa1df9, 4eb1ff2a629e48298e14b6e824a66fa7; no voters:
I20260928 02:46:25.774740 2108 raft_consensus.cc:2454] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49dccc1c4dd5432a8a74255a98fa1df9 in term 0.
I20260928 02:46:25.774904 2309 raft_consensus.cc:2781] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:25.775032 2309 raft_consensus.cc:482] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:25.775115 2309 raft_consensus.cc:3037] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:25.776453 2309 raft_consensus.cc:504] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.776994 2309 leader_election.cc:288] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb1ff2a629e48298e14b6e824a66fa7 (127.28.248.131:38233), 2597748504804b3c872c707804f5ea1b (127.28.248.130:36319)
I20260928 02:46:25.777415 2239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ae9bf4e0dc45b891c16ea25a2a2e04" candidate_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb1ff2a629e48298e14b6e824a66fa7"
I20260928 02:46:25.777501 2108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ae9bf4e0dc45b891c16ea25a2a2e04" candidate_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2597748504804b3c872c707804f5ea1b"
I20260928 02:46:25.777582 2239 raft_consensus.cc:3037] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:25.777621 2108 raft_consensus.cc:3037] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:25.779093 2239 raft_consensus.cc:2454] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49dccc1c4dd5432a8a74255a98fa1df9 in term 1.
I20260928 02:46:25.779093 2108 raft_consensus.cc:2454] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49dccc1c4dd5432a8a74255a98fa1df9 in term 1.
I20260928 02:46:25.779470 1911 leader_election.cc:302] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2597748504804b3c872c707804f5ea1b, 49dccc1c4dd5432a8a74255a98fa1df9; no voters:
I20260928 02:46:25.779677 2309 raft_consensus.cc:2781] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:25.780010 2309 raft_consensus.cc:686] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 1 LEADER]: Becoming Leader. State: Replica: 49dccc1c4dd5432a8a74255a98fa1df9, State: Running, Role: LEADER
I20260928 02:46:25.780377 2309 consensus_queue.cc:237] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } }
I20260928 02:46:25.784008 1833 catalog_manager.cc:5495] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49dccc1c4dd5432a8a74255a98fa1df9 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49dccc1c4dd5432a8a74255a98fa1df9" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:25.794320 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.792706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.28.248.129:46343)
0928 02:46:25.792854 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:25.792858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:25.792872 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:25.793001 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:25.793004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:25.793068 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:25.793139 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:25.793924 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:25.793927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:25.793930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:25.793944 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:25.793969 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:25.793979 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:25.794080 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:25.794137 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:46:25.794180 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:25.801170 2108 raft_consensus.cc:1261] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 1 FOLLOWER]: Refusing update from remote peer 49dccc1c4dd5432a8a74255a98fa1df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:25.801195 2239 raft_consensus.cc:1261] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 1 FOLLOWER]: Refusing update from remote peer 49dccc1c4dd5432a8a74255a98fa1df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:25.802026 2313 consensus_queue.cc:1035] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2597748504804b3c872c707804f5ea1b" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 36319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:25.802256 2309 consensus_queue.cc:1035] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb1ff2a629e48298e14b6e824a66fa7" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:25.811450 2317 mvcc.cc:205] Tried to move back new op lower bound from 7334148447432994816 to 7334148447359385600. Current Snapshot: MvccSnapshot[applied={T|T < 7334148447432994816}]
I20260928 02:46:25.821712 1833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:25.821957 1833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:25.824985 1833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 744cc1c783d64ec68b0aa443d366878a: Sending DeleteTablet for 3 replicas of tablet 72ae9bf4e0dc45b891c16ea25a2a2e04
I20260928 02:46:25.826124 2219 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:25 UTC) from {username='slave'} at 127.0.0.1:41332
I20260928 02:46:25.826184 1957 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:25 UTC) from {username='slave'} at 127.0.0.1:35500
I20260928 02:46:25.826177 2088 tablet_service.cc:1514] Processing DeleteTablet for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:25 UTC) from {username='slave'} at 127.0.0.1:54630
I20260928 02:46:25.826826 2329 tablet_replica.cc:330] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: stopping tablet replica
I20260928 02:46:25.826933 2327 tablet_replica.cc:330] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9: stopping tablet replica
I20260928 02:46:25.827296 2327 raft_consensus.cc:2229] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:25.827297 2329 raft_consensus.cc:2229] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:25.827569 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1895
I20260928 02:46:25.827695 2327 raft_consensus.cc:2258] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 49dccc1c4dd5432a8a74255a98fa1df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:25.827728 2329 raft_consensus.cc:2258] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:25.828133 2328 tablet_replica.cc:330] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: stopping tablet replica
I20260928 02:46:25.828503 2328 raft_consensus.cc:2229] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:25.828871 2328 raft_consensus.cc:2258] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:25.830112 2329 ts_tablet_manager.cc:1905] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:25.830470 2328 ts_tablet_manager.cc:1905] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:25.832504 2329 ts_tablet_manager.cc:1918] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:25.832665 2329 log.cc:1196] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/72ae9bf4e0dc45b891c16ea25a2a2e04
I20260928 02:46:25.832962 2329 ts_tablet_manager.cc:1939] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 2597748504804b3c872c707804f5ea1b: Deleting consensus metadata
I20260928 02:46:25.834033 2328 ts_tablet_manager.cc:1918] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:25.834139 2328 log.cc:1196] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/72ae9bf4e0dc45b891c16ea25a2a2e04
I20260928 02:46:25.834309 1820 catalog_manager.cc:4841] TS 2597748504804b3c872c707804f5ea1b (127.28.248.130:36319): tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 (table test-table [id=180fa49d0ccf452d87e5bb180c335c29]) successfully deleted
I20260928 02:46:25.834465 2328 ts_tablet_manager.cc:1939] T 72ae9bf4e0dc45b891c16ea25a2a2e04 P 4eb1ff2a629e48298e14b6e824a66fa7: Deleting consensus metadata
I20260928 02:46:25.835889 1819 catalog_manager.cc:4841] TS 4eb1ff2a629e48298e14b6e824a66fa7 (127.28.248.131:38233): tablet 72ae9bf4e0dc45b891c16ea25a2a2e04 (table test-table [id=180fa49d0ccf452d87e5bb180c335c29]) successfully deleted
W20260928 02:46:25.836450 1820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:46343 (error 108)
I20260928 02:46:25.836479 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2026
I20260928 02:46:25.838779 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.838377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:46343 (local address 127.0.0.1:35510)
0928 02:46:25.838565 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.838662 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:46:25.838999 1820 catalog_manager.cc:4584] TS 49dccc1c4dd5432a8a74255a98fa1df9 (127.28.248.129:46343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72ae9bf4e0dc45b891c16ea25a2a2e04: Network error: Client connection negotiation failed: client connection to 127.28.248.129:46343: connect: Connection refused (error 111)
I20260928 02:46:25.842665 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2157
I20260928 02:46:25.849581 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 1804
2026-09-28T02:46:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1100 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:46:25.872140 29666 mini_oidc.cc:152] Starting JWKS server
I20260928 02:46:25.873806 29666 webserver.cc:460] Webserver started at https://127.0.0.1:38613/ using document root <none> and password file <none>
I20260928 02:46:25.874004 29666 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:46:25.874289 29666 webserver.cc:460] Webserver started at http://127.0.0.1:33435/ using document root <none> and password file <none>
2026-09-28T02:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:25Z Disabled control of system clock
I20260928 02:46:25.883419 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39003
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:39003
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38613/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/testchainca.cert with env {}
W20260928 02:46:25.992556 2336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:25.992873 2336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:25.992916 2336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:25.997031 2336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:25.997102 2336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:25.997117 2336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:25.997135 2336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:25.997150 2336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:46:25.997218 2336 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:46:25.997237 2336 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38613/jwks.json
I20260928 02:46:26.002897 2336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:39003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38613/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:26.004051 2336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:26.005169 2336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:46:26.014658 2336 server_base.cc:1034] running on GCE node
W20260928 02:46:26.014643 2341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.014613 2342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.014613 2344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:26.015581 2336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:26.016834 2336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:26.018006 2336 hybrid_clock.cc:648] HybridClock initialized: now 1790563586017972 us; error 48 us; skew 500 ppm
I20260928 02:46:26.020217 2336 webserver.cc:460] Webserver started at http://127.28.248.190:34735/ using document root <none> and password file <none>
I20260928 02:46:26.020885 2336 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:26.020951 2336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:26.021171 2336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:26.023343 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "401614e8059f4b18989bb4b4ddd5487c"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "aace27e4279d248c2e566f2fdc9c80b1"
server_key_iv: "83ce918630bf3cf122c98c2c30c6dbeb"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.023916 2336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "401614e8059f4b18989bb4b4ddd5487c"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "aace27e4279d248c2e566f2fdc9c80b1"
server_key_iv: "83ce918630bf3cf122c98c2c30c6dbeb"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.027745 2336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:26.030711 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:26.032130 2336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:46:26.032290 2336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "401614e8059f4b18989bb4b4ddd5487c"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "aace27e4279d248c2e566f2fdc9c80b1"
server_key_iv: "83ce918630bf3cf122c98c2c30c6dbeb"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.032411 2336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:26.069497 2336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:26.070288 2336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:26.070535 2336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:26.078064 2336 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:39003
I20260928 02:46:26.078075 2402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:39003 every 8 connection(s)
I20260928 02:46:26.079098 2336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:26.079623 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2336
I20260928 02:46:26.079789 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:26.080101 29666 external_mini_cluster.cc:1428] Setting key 80e40dce0db70ea6047c4505f6b6aa9b
I20260928 02:46:26.082907 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:26.089906 2403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c: Bootstrap starting.
I20260928 02:46:26.090937 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.081598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.28.248.190:39003)
0928 02:46:26.082307 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.082327 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.082371 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:46:26.083245 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.083268 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.083770 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.084142 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:26.084634 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.086091 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.087004 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.087690 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.087867 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.088901 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:26.088907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:26.088924 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:26.089429 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:26.089499 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:26.089524 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:26.089626 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:26.089730 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:46:26.090006 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":108,"threads_started":1}
I20260928 02:46:26.092522 2403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:26.093439 2403 log.cc:824] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:26.094208 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:46:26.095896 2403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c: No bootstrap required, opened a new log
I20260928 02:46:26.099575 2403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401614e8059f4b18989bb4b4ddd5487c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39003 } }
I20260928 02:46:26.099812 2403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:26.099890 2403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401614e8059f4b18989bb4b4ddd5487c, State: Initialized, Role: FOLLOWER
I20260928 02:46:26.100304 2403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401614e8059f4b18989bb4b4ddd5487c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39003 } }
I20260928 02:46:26.100447 2403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:26.100512 2403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:26.100656 2403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:26.101609 2403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401614e8059f4b18989bb4b4ddd5487c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39003 } }
I20260928 02:46:26.101976 2403 leader_election.cc:302] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401614e8059f4b18989bb4b4ddd5487c; no voters:
I20260928 02:46:26.102305 2403 leader_election.cc:288] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:26.102455 2408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:26.102707 2408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [term 1 LEADER]: Becoming Leader. State: Replica: 401614e8059f4b18989bb4b4ddd5487c, State: Running, Role: LEADER
I20260928 02:46:26.103021 2408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401614e8059f4b18989bb4b4ddd5487c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39003 } }
I20260928 02:46:26.103556 2403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:26.104782 2410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401614e8059f4b18989bb4b4ddd5487c. Latest consensus state: current_term: 1 leader_uuid: "401614e8059f4b18989bb4b4ddd5487c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401614e8059f4b18989bb4b4ddd5487c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39003 } } }
I20260928 02:46:26.104818 2409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401614e8059f4b18989bb4b4ddd5487c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401614e8059f4b18989bb4b4ddd5487c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39003 } } }
I20260928 02:46:26.104938 2409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.104938 2410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.105396 2417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:26.108819 2417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:26.114451 2417 catalog_manager.cc:1347] Generated new cluster ID: 35dab25a2aa94601a578c24f114a554c
I20260928 02:46:26.114528 2417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:26.124095 2417 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:26.125154 2417 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:26.132580 2417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401614e8059f4b18989bb4b4ddd5487c: Generated new TSK 0
I20260928 02:46:26.133363 2417 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:46:26.165608 2433 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:46:26.165741 2433 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:46:26.166417 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.147526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.28.248.190:39003)
0928 02:46:26.147683 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.147688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.147935 (+ 247us) server_negotiation.cc:407] Connection header received
0928 02:46:26.148174 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.148178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.148241 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.148363 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:46:26.148751 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.149598 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.150672 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.151023 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.151126 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.151542 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:46:26.165975 (+ 14433us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:46:26.166269 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 02:46:26.166484 2406 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:46:26.167591 2432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:26.147426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39003 (local address 127.0.0.1:38498)
0928 02:46:26.147834 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:26.147850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:26.147985 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:26.148412 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:26.148414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:26.148424 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:46:26.148541 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:26.148548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.149757 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.149762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:26.150540 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:26.150550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.151155 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.151159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:26.151258 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.151315 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:46:26.166221 (+ 14906us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:46:26.166439 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.248.190:39003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":141,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.248.190:39003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:46:26.168558 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2336
2026-09-28T02:46:26Z chronyd exiting
I20260928 02:46:26.206137 29666 test_util.cc:175] -----------------------------------------------
I20260928 02:46:26.206259 29666 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563560542449-29666-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:46:26.207304 29666 mini_oidc.cc:152] Starting JWKS server
I20260928 02:46:26.208184 29666 webserver.cc:460] Webserver started at https://127.0.0.1:41479/ using document root <none> and password file <none>
I20260928 02:46:26.208312 29666 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:46:26.208617 29666 webserver.cc:460] Webserver started at http://127.0.0.1:33787/ using document root <none> and password file <none>
2026-09-28T02:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:26Z Disabled control of system clock
I20260928 02:46:26.217753 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:32907
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:32907
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41479/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/testchainca.cert with env {}
W20260928 02:46:26.332015 2439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:26.332307 2439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:26.332350 2439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:26.336416 2439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:26.336489 2439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:26.336506 2439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:26.336522 2439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:26.336539 2439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:46:26.336555 2439 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:46:26.336642 2439 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41479/jwks.json
I20260928 02:46:26.342298 2439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:32907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41479/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:26.343433 2439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:26.344485 2439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:26.353354 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.353354 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.353462 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:26.353637 2439 server_base.cc:1034] running on GCE node
I20260928 02:46:26.354336 2439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:26.355451 2439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:26.356647 2439 hybrid_clock.cc:648] HybridClock initialized: now 1790563586356596 us; error 26 us; skew 500 ppm
I20260928 02:46:26.358562 2439 webserver.cc:460] Webserver started at http://127.28.248.190:42761/ using document root <none> and password file <none>
I20260928 02:46:26.359138 2439 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:26.359200 2439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:26.359411 2439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:26.361567 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "d12b9b48ac56442bad1140cda586d323"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "a77a09ecac134678bb08d2f93ea60984"
server_key_iv: "65875119702d4dc1666391b2a9e0de33"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.362124 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "d12b9b48ac56442bad1140cda586d323"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "a77a09ecac134678bb08d2f93ea60984"
server_key_iv: "65875119702d4dc1666391b2a9e0de33"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.365849 2439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:26.368299 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:26.369630 2439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:26.369731 2439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "d12b9b48ac56442bad1140cda586d323"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "a77a09ecac134678bb08d2f93ea60984"
server_key_iv: "65875119702d4dc1666391b2a9e0de33"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.369820 2439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:26.380561 2439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:26.381305 2439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:26.381469 2439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:26.388427 2439 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:32907
I20260928 02:46:26.388409 2505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:32907 every 8 connection(s)
I20260928 02:46:26.389434 2439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:26.392558 2506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:26.393698 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2439
I20260928 02:46:26.393841 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:26.394150 29666 external_mini_cluster.cc:1428] Setting key 8d5023c686396c529122f8d3148c23ae
I20260928 02:46:26.398753 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323: Bootstrap starting.
I20260928 02:46:26.400946 2506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:26.401691 2506 log.cc:824] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:26.403714 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323: No bootstrap required, opened a new log
I20260928 02:46:26.403780 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.395538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.28.248.190:32907)
0928 02:46:26.396166 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.396181 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.396216 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:46:26.396869 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.396885 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.397277 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.397614 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:26.398046 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.399157 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.400141 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.400665 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.400920 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.401869 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:26.401874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:26.401887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:26.402418 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:26.402494 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:26.402524 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:26.402644 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:26.402824 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:46:26.403086 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260928 02:46:26.406716 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:46:26.407822 2506 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12b9b48ac56442bad1140cda586d323" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32907 } }
I20260928 02:46:26.408058 2506 raft_consensus.cc:374] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:26.408134 2506 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12b9b48ac56442bad1140cda586d323, State: Initialized, Role: FOLLOWER
I20260928 02:46:26.408666 2506 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12b9b48ac56442bad1140cda586d323" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32907 } }
I20260928 02:46:26.408824 2506 raft_consensus.cc:388] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:26.408893 2506 raft_consensus.cc:482] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:26.408983 2506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:26.409930 2506 raft_consensus.cc:504] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12b9b48ac56442bad1140cda586d323" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32907 } }
I20260928 02:46:26.410271 2506 leader_election.cc:302] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d12b9b48ac56442bad1140cda586d323; no voters:
I20260928 02:46:26.410534 2506 leader_election.cc:288] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:26.410666 2511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:26.410898 2511 raft_consensus.cc:686] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [term 1 LEADER]: Becoming Leader. State: Replica: d12b9b48ac56442bad1140cda586d323, State: Running, Role: LEADER
I20260928 02:46:26.411190 2511 consensus_queue.cc:237] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12b9b48ac56442bad1140cda586d323" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32907 } }
I20260928 02:46:26.411761 2506 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:26.412873 2513 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d12b9b48ac56442bad1140cda586d323. Latest consensus state: current_term: 1 leader_uuid: "d12b9b48ac56442bad1140cda586d323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12b9b48ac56442bad1140cda586d323" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32907 } } }
I20260928 02:46:26.412995 2513 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.413251 2512 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d12b9b48ac56442bad1140cda586d323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d12b9b48ac56442bad1140cda586d323" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32907 } } }
I20260928 02:46:26.413348 2512 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.413473 2520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:26.416405 2520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:26.422173 2520 catalog_manager.cc:1347] Generated new cluster ID: 4f09384076f94b99a04cbea9e9cd0428
I20260928 02:46:26.422272 2520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:26.430227 2520 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:26.431088 2520 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:26.436636 2520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d12b9b48ac56442bad1140cda586d323: Generated new TSK 0
I20260928 02:46:26.437330 2520 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:46:26.461174 2536 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:46:26.461282 2536 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:46:26.462080 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.446400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.28.248.190:32907)
0928 02:46:26.446523 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.446529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.446705 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:46:26.446901 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.446903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.446950 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.447062 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:46:26.447299 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.447847 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.448766 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.448987 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.449096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.449398 (+ 302us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:46:26.461582 (+ 12184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:46:26.461919 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:46:26.462157 2509 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:46:26.462185 2535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:26.446321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:32907 (local address 127.0.0.1:40660)
0928 02:46:26.446611 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:26.446623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:26.446754 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:26.447060 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:26.447063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:26.447071 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:46:26.447180 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:26.447188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.447971 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.447976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:26.448632 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:26.448662 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.449103 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.449106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:26.449192 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.449222 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:46:26.461828 (+ 12606us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:46:26.461996 (+ 168us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.248.190:32907: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41479/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260928 02:46:26.463080 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2439
2026-09-28T02:46:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:46:26.503064 29666 mini_oidc.cc:152] Starting JWKS server
I20260928 02:46:26.504019 29666 webserver.cc:460] Webserver started at https://127.0.0.1:43025/ using document root <none> and password file <none>
I20260928 02:46:26.504230 29666 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:46:26.504652 29666 webserver.cc:460] Webserver started at http://127.0.0.1:36635/ using document root <none> and password file <none>
2026-09-28T02:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:26Z Disabled control of system clock
I20260928 02:46:26.513307 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43793
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:43487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:43793
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43025/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:46:26.618696 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:26.619012 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:26.619088 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:26.623077 2542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:26.623168 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:26.623198 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:26.623237 2542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:26.623270 2542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:46:26.623304 2542 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:46:26.623324 2542 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43025/jwks.json
I20260928 02:46:26.627996 2542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:43793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43025/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:26.629122 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:26.630520 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:26.638749 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.638728 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.638741 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:26.638983 2542 server_base.cc:1034] running on GCE node
I20260928 02:46:26.639621 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:26.640736 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:26.641891 2542 hybrid_clock.cc:648] HybridClock initialized: now 1790563586641862 us; error 44 us; skew 500 ppm
I20260928 02:46:26.643615 2542 webserver.cc:460] Webserver started at http://127.28.248.190:39997/ using document root <none> and password file <none>
I20260928 02:46:26.644167 2542 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:26.644227 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:26.644479 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:26.646525 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "a2a69d48a1d94de891e75cabfcb850f6"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "723bf15bd1487dbf6dfb09ef49e99f2b"
server_key_iv: "d0d31f501a616feff00f489ca8b2ed22"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.647070 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "a2a69d48a1d94de891e75cabfcb850f6"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "723bf15bd1487dbf6dfb09ef49e99f2b"
server_key_iv: "d0d31f501a616feff00f489ca8b2ed22"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.650713 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:26.653172 2556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:26.654372 2542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:26.654513 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "a2a69d48a1d94de891e75cabfcb850f6"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "723bf15bd1487dbf6dfb09ef49e99f2b"
server_key_iv: "d0d31f501a616feff00f489ca8b2ed22"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.654628 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:26.672588 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:26.673341 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:26.673544 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:26.680352 2542 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:43793
I20260928 02:46:26.680373 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:43793 every 8 connection(s)
I20260928 02:46:26.681337 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:26.684387 2609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:26.688874 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2542
I20260928 02:46:26.688997 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:26.689289 29666 external_mini_cluster.cc:1428] Setting key 5811db71fb62579547d123c563c3b501
I20260928 02:46:26.690121 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6: Bootstrap starting.
I20260928 02:46:26.692323 2609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:26.693214 2609 log.cc:824] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:26.695283 2609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6: No bootstrap required, opened a new log
I20260928 02:46:26.698578 2609 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a69d48a1d94de891e75cabfcb850f6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43793 } }
I20260928 02:46:26.698799 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.690616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.28.248.190:43793)
0928 02:46:26.691207 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.691223 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.691257 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:26.691856 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.691871 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.692262 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.692590 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:26.692995 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.694182 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.695316 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.695807 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.695927 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.696775 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:26.696780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:26.696792 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:26.697166 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:26.697217 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:26.697235 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:26.697309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:26.697474 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:46:26.697751 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":126,"threads_started":1}
I20260928 02:46:26.698804 2609 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:26.698966 2609 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a69d48a1d94de891e75cabfcb850f6, State: Initialized, Role: FOLLOWER
I20260928 02:46:26.699435 2609 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a69d48a1d94de891e75cabfcb850f6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43793 } }
I20260928 02:46:26.699570 2609 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:26.699620 2609 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:26.699697 2609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:26.700527 2609 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a69d48a1d94de891e75cabfcb850f6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43793 } }
I20260928 02:46:26.700876 2609 leader_election.cc:302] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a69d48a1d94de891e75cabfcb850f6; no voters:
I20260928 02:46:26.701189 2609 leader_election.cc:288] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:26.701273 2614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:26.701442 2614 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [term 1 LEADER]: Becoming Leader. State: Replica: a2a69d48a1d94de891e75cabfcb850f6, State: Running, Role: LEADER
I20260928 02:46:26.701709 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:46:26.701713 2614 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a69d48a1d94de891e75cabfcb850f6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43793 } }
I20260928 02:46:26.702378 2609 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:26.702805 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a69d48a1d94de891e75cabfcb850f6. Latest consensus state: current_term: 1 leader_uuid: "a2a69d48a1d94de891e75cabfcb850f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a69d48a1d94de891e75cabfcb850f6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43793 } } }
I20260928 02:46:26.702920 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.703291 2618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:26.704033 2615 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a69d48a1d94de891e75cabfcb850f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a69d48a1d94de891e75cabfcb850f6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43793 } } }
I20260928 02:46:26.704180 2615 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.706405 2618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:26.711840 2618 catalog_manager.cc:1347] Generated new cluster ID: 920ff503e6da4391a212364df58228cd
I20260928 02:46:26.711911 2618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:26.719929 2618 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:26.721153 2618 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:26.729045 2618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a69d48a1d94de891e75cabfcb850f6: Generated new TSK 0
I20260928 02:46:26.729683 2618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:26.747779 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.742806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.28.248.190:43793)
0928 02:46:26.742972 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.742977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.743120 (+ 143us) server_negotiation.cc:407] Connection header received
0928 02:46:26.743271 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.743273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.743317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.743405 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:46:26.743669 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.744197 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.745462 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.745672 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.745749 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.746063 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:46:26.747439 (+ 1376us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:46:26.747654 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:46:26.747844 2612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:46:26.749717 2638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:26.742747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43793 (local address 127.0.0.1:42934)
0928 02:46:26.743028 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:26.743043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:26.743154 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:26.743431 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:26.743434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:26.743442 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:46:26.743561 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:26.743569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.744319 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.744323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:26.745354 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:26.745364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.745787 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.745791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:26.745878 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.745907 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:46:26.747599 (+ 1692us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:46:26.749538 (+ 1939us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.248.190:43793: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1749,"thread_start_us":110,"threads_started":1}
I20260928 02:46:26.750257 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2542
E20260928 02:46:26.760809 2639 webserver.cc:533] Webserver: sslize failed:
E20260928 02:46:26.760878 2639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T02:46:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:26Z Disabled control of system clock
I20260928 02:46:26.798544 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:40153
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:40153
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/testchainca.cert with env {}
W20260928 02:46:26.907060 2643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:26.907307 2643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:26.907346 2643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:26.911351 2643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:26.911404 2643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:26.911420 2643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:26.911437 2643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:26.911453 2643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:46:26.911469 2643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:46:26.911489 2643 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:46:26.916652 2643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:40153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:26.917613 2643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:26.918524 2643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:26.926676 2649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.926672 2648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:26.926677 2651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:26.927030 2643 server_base.cc:1034] running on GCE node
I20260928 02:46:26.927587 2643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:26.928682 2643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:26.929863 2643 hybrid_clock.cc:648] HybridClock initialized: now 1790563586929857 us; error 30 us; skew 500 ppm
I20260928 02:46:26.931603 2643 webserver.cc:460] Webserver started at http://127.28.248.190:40711/ using document root <none> and password file <none>
I20260928 02:46:26.932165 2643 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:26.932253 2643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:26.932480 2643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:26.934487 2643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "52043912ccdf486f8e067262b70ef1ab"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "b9b52a4dfba844dd4f41d9b6bb521f0d"
server_key_iv: "e832d39ee3af060531342ff7d75d575b"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.935060 2643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "52043912ccdf486f8e067262b70ef1ab"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "b9b52a4dfba844dd4f41d9b6bb521f0d"
server_key_iv: "e832d39ee3af060531342ff7d75d575b"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.938618 2643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:26.940861 2657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:26.942041 2643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:26.942171 2643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "52043912ccdf486f8e067262b70ef1ab"
format_stamp: "Formatted at 2026-09-28 02:46:26 on dist-test-slave-r238"
server_key: "b9b52a4dfba844dd4f41d9b6bb521f0d"
server_key_iv: "e832d39ee3af060531342ff7d75d575b"
server_key_version: "encryptionkey@0"
I20260928 02:46:26.942274 2643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:26.969213 2643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:26.969899 2643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:26.970093 2643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:26.976737 2643 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:40153
I20260928 02:46:26.976764 2709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:40153 every 8 connection(s)
I20260928 02:46:26.977648 2643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:26.980367 2710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:26.983909 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2643
I20260928 02:46:26.984037 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:26.984326 29666 external_mini_cluster.cc:1428] Setting key 939f0067d1826ef7656bf39c91783527
I20260928 02:46:26.986462 2710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab: Bootstrap starting.
I20260928 02:46:26.988692 2710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:26.989419 2710 log.cc:824] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:26.991298 2710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab: No bootstrap required, opened a new log
I20260928 02:46:26.993845 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.985750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.28.248.190:40153)
0928 02:46:26.986371 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:26.986386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:26.986424 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:46:26.987103 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:26.987122 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:26.987518 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:26.987859 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:26.988298 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.989401 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.990265 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:26.990802 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:26.990994 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:26.991844 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:26.991856 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:26.991870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:26.992251 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:26.992304 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:26.992323 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:26.992436 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:26.992592 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:46:26.992879 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260928 02:46:26.994496 2710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52043912ccdf486f8e067262b70ef1ab" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40153 } }
I20260928 02:46:26.994738 2710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:26.994824 2710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52043912ccdf486f8e067262b70ef1ab, State: Initialized, Role: FOLLOWER
I20260928 02:46:26.995281 2710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52043912ccdf486f8e067262b70ef1ab" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40153 } }
I20260928 02:46:26.995416 2710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:26.995468 2710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:26.995549 2710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:26.996654 2710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52043912ccdf486f8e067262b70ef1ab" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40153 } }
I20260928 02:46:26.996893 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:46:26.997028 2710 leader_election.cc:302] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52043912ccdf486f8e067262b70ef1ab; no voters:
I20260928 02:46:26.997378 2710 leader_election.cc:288] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:26.997465 2715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:26.997622 2715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [term 1 LEADER]: Becoming Leader. State: Replica: 52043912ccdf486f8e067262b70ef1ab, State: Running, Role: LEADER
I20260928 02:46:26.997906 2715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52043912ccdf486f8e067262b70ef1ab" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40153 } }
I20260928 02:46:26.998622 2710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:26.999089 2715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52043912ccdf486f8e067262b70ef1ab. Latest consensus state: current_term: 1 leader_uuid: "52043912ccdf486f8e067262b70ef1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52043912ccdf486f8e067262b70ef1ab" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40153 } } }
I20260928 02:46:26.999202 2715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:26.999567 2716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52043912ccdf486f8e067262b70ef1ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52043912ccdf486f8e067262b70ef1ab" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40153 } } }
I20260928 02:46:26.999743 2716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:27.000057 2719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:27.003193 2719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:27.009315 2719 catalog_manager.cc:1347] Generated new cluster ID: 32aa5645d2874b968309a08c74f2344e
I20260928 02:46:27.009402 2719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:27.015699 2719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:27.016585 2719 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:27.021940 2719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52043912ccdf486f8e067262b70ef1ab: Generated new TSK 0
I20260928 02:46:27.022840 2719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:27.092496 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:27.034780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.28.248.190:40153)
0928 02:46:27.034913 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:27.034918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:27.035178 (+ 260us) server_negotiation.cc:407] Connection header received
0928 02:46:27.035348 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:27.035351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:27.035399 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:27.035487 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:46:27.035736 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:27.036276 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:27.037521 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:27.037734 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:27.037810 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:27.038147 (+ 337us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:46:27.092077 (+ 53930us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:46:27.092370 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36496: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 02:46:27.092682 2713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36496: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:46:27.092793 2739 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:27.034685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:40153 (local address 127.0.0.1:36496)
0928 02:46:27.035082 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:27.035095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:27.035208 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:27.035510 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:27.035513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:27.035521 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:46:27.035631 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:27.035638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:27.036395 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:27.036399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:27.037409 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:27.037420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:27.037851 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:27.037855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:27.037948 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:27.037985 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:46:27.092342 (+ 54357us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:46:27.092560 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.248.190:40153: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":103,"threads_started":1}
I20260928 02:46:27.093474 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2643
2026-09-28T02:46:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (321 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:27 dist-test-slave-r238 krb5kdc[2751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:27 dist-test-slave-r238 krb5kdc[2751](info): set up 2 sockets
Sep 28 02:46:27 dist-test-slave-r238 krb5kdc[2751](info): commencing operation
krb5kdc: starting...
W20260928 02:46:28.377154 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:28 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:28.503746 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41947
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:41947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:28.616498 2764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:28.616884 2764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:28.616930 2764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:28.621009 2764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:28.621075 2764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:28.621095 2764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:28.621112 2764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:28.621127 2764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:28.626601 2764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:41947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:28.627735 2764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:28.628863 2764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:28.637835 2772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:28.637825 2769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:28.637825 2770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:28.637892 2764 server_base.cc:1034] running on GCE node
I20260928 02:46:28.638643 2764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:28.639767 2764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:28.640928 2764 hybrid_clock.cc:648] HybridClock initialized: now 1790563588640913 us; error 27 us; skew 500 ppm
Sep 28 02:46:28 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563588, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:28.643680 2764 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:28.644550 2764 webserver.cc:460] Webserver started at http://127.28.248.190:42593/ using document root <none> and password file <none>
I20260928 02:46:28.645126 2764 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:28.645218 2764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:28.645462 2764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:28.647501 2764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "6b7c50bc21824c0fbd94c238a027406a"
format_stamp: "Formatted at 2026-09-28 02:46:28 on dist-test-slave-r238"
server_key: "1632f3e94a2fd201d5b2e134f8fe89de"
server_key_iv: "7aa6c7db483efeea1c958fd890dfcf96"
server_key_version: "encryptionkey@0"
I20260928 02:46:28.648100 2764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "6b7c50bc21824c0fbd94c238a027406a"
format_stamp: "Formatted at 2026-09-28 02:46:28 on dist-test-slave-r238"
server_key: "1632f3e94a2fd201d5b2e134f8fe89de"
server_key_iv: "7aa6c7db483efeea1c958fd890dfcf96"
server_key_version: "encryptionkey@0"
I20260928 02:46:28.652182 2764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:28.655061 2779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:28.656322 2764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:46:28.656491 2764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "6b7c50bc21824c0fbd94c238a027406a"
format_stamp: "Formatted at 2026-09-28 02:46:28 on dist-test-slave-r238"
server_key: "1632f3e94a2fd201d5b2e134f8fe89de"
server_key_iv: "7aa6c7db483efeea1c958fd890dfcf96"
server_key_version: "encryptionkey@0"
I20260928 02:46:28.656667 2764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:28.673344 2764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:28.676373 2764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:28.676592 2764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:28.683751 2764 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:41947
I20260928 02:46:28.683892 2831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:41947 every 8 connection(s)
I20260928 02:46:28.684835 2764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:28.687685 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:28.690575 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2764
I20260928 02:46:28.690735 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:28.691102 29666 external_mini_cluster.cc:1428] Setting key 3c18d9c36005f82bff98cb1ed2d4a3f4
I20260928 02:46:28.694543 2832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a: Bootstrap starting.
I20260928 02:46:28.697849 2832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:28.698843 2832 log.cc:824] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:28.701556 2832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a: No bootstrap required, opened a new log
Sep 28 02:46:28 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:28.705912 2832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c50bc21824c0fbd94c238a027406a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41947 } }
I20260928 02:46:28.706208 2832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:28.706261 2832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7c50bc21824c0fbd94c238a027406a, State: Initialized, Role: FOLLOWER
I20260928 02:46:28.706825 2832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c50bc21824c0fbd94c238a027406a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41947 } }
I20260928 02:46:28.706984 2832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:28.707050 2832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:28.707146 2832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:28.708354 2832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c50bc21824c0fbd94c238a027406a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41947 } }
I20260928 02:46:28.708753 2832 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b7c50bc21824c0fbd94c238a027406a; no voters:
I20260928 02:46:28.709004 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:28.692738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.28.248.190:41947)
0928 02:46:28.693244 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:28.693261 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:28.693296 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:46:28.693902 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:28.693921 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:28.694256 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:28.694615 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:28.695502 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:28.697228 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:28.698614 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:28.699462 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:28.699643 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:28.702764 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:28.702770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:28.702790 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:28.702822 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:28.705534 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:28.706088 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:28.706094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:28.706100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:28.706246 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:28.706587 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:28.706591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:28.706593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:28.707128 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:28.707326 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:28.707608 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:46:28.707934 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":167,"threads_started":1}
I20260928 02:46:28.709093 2832 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:28.709245 2837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:28.709482 2837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [term 1 LEADER]: Becoming Leader. State: Replica: 6b7c50bc21824c0fbd94c238a027406a, State: Running, Role: LEADER
I20260928 02:46:28.709810 2837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c50bc21824c0fbd94c238a027406a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41947 } }
I20260928 02:46:28.710350 2832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:28.711062 2837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b7c50bc21824c0fbd94c238a027406a. Latest consensus state: current_term: 1 leader_uuid: "6b7c50bc21824c0fbd94c238a027406a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c50bc21824c0fbd94c238a027406a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41947 } } }
I20260928 02:46:28.711176 2837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:28.711804 2845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:28.712083 2838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b7c50bc21824c0fbd94c238a027406a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b7c50bc21824c0fbd94c238a027406a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41947 } } }
I20260928 02:46:28.712203 2838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:28.715154 2845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:28.720908 2845 catalog_manager.cc:1347] Generated new cluster ID: cf93d378f42348199857599f78263234
I20260928 02:46:28.721055 2845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:28.755920 2845 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:28.756973 2845 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:28.768231 2845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b7c50bc21824c0fbd94c238a027406a: Generated new TSK 0
I20260928 02:46:28.769193 2845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:28.805052 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41947
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:28.920958 2858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:28.921234 2858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:28.921279 2858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:28.925534 2858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:28.925608 2858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:28.925702 2858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:28.931262 2858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:28.932474 2858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:28.933919 2858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:28.943828 2864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:28.943885 2866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:28.943830 2863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:28.944165 2858 server_base.cc:1034] running on GCE node
I20260928 02:46:28.944778 2858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:28.945547 2858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:28.946805 2858 hybrid_clock.cc:648] HybridClock initialized: now 1790563588946766 us; error 53 us; skew 500 ppm
Sep 28 02:46:28 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563588, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:28.955988 2858 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:28.957239 2858 webserver.cc:460] Webserver started at http://127.28.248.129:39465/ using document root <none> and password file <none>
I20260928 02:46:28.957983 2858 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:28.958125 2858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:28.958375 2858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:28.961133 2858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "427a3da47f214912a622d16f38d5da07"
format_stamp: "Formatted at 2026-09-28 02:46:28 on dist-test-slave-r238"
server_key: "09abc1698f28795daa6109de50ae9eea"
server_key_iv: "ae48785c520ff2827d162b6b4872de57"
server_key_version: "encryptionkey@0"
I20260928 02:46:28.961760 2858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "427a3da47f214912a622d16f38d5da07"
format_stamp: "Formatted at 2026-09-28 02:46:28 on dist-test-slave-r238"
server_key: "09abc1698f28795daa6109de50ae9eea"
server_key_iv: "ae48785c520ff2827d162b6b4872de57"
server_key_version: "encryptionkey@0"
I20260928 02:46:28.965929 2858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:46:28.968515 2873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:28.969882 2858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:28.970037 2858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "427a3da47f214912a622d16f38d5da07"
format_stamp: "Formatted at 2026-09-28 02:46:28 on dist-test-slave-r238"
server_key: "09abc1698f28795daa6109de50ae9eea"
server_key_iv: "ae48785c520ff2827d162b6b4872de57"
server_key_version: "encryptionkey@0"
I20260928 02:46:28.970158 2858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:28.984081 2858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:28.987471 2858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:28.987699 2858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:28.988332 2858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:28.989238 2858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:28.989310 2858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:28.989383 2858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:28.989426 2858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:28.999131 2858 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:43579
I20260928 02:46:28.999148 2986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:43579 every 8 connection(s)
I20260928 02:46:29.000088 2858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:29.002027 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2858
I20260928 02:46:29.002144 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:29.002429 29666 external_mini_cluster.cc:1428] Setting key 2381eb43a5025377804b23f47a84b4c0
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563588, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:29.017050 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.001954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:35531 (local address 127.28.248.190:41947)
0928 02:46:29.002119 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.002124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.003193 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 02:46:29.004800 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.004803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.004866 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.004951 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.006838 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.007383 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.009371 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.009611 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.009705 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.013068 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:29.013071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:29.013073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:29.013099 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:29.014766 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.015278 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.015281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.015282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.015359 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.015708 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.015710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.015712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.015943 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:29.016020 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:29.016636 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 02:46:29.016796 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:29.017908 2989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.002483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41947 (local address 127.28.248.129:35531)
0928 02:46:29.003038 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:29.003099 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:29.004526 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:29.005143 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:29.005150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:29.005694 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:29.006462 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:29.006485 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.007522 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.007529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:29.009221 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:29.009236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.009792 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.009798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:29.010111 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.010772 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:29.010795 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:29.012889 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:29.014915 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:29.014920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:29.014933 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:29.015192 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:29.015451 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:29.015454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:29.015455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:29.015593 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:29.016046 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:29.016056 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:29.016336 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 02:46:29.016505 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 02:46:29.016912 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":131,"threads_started":1}
I20260928 02:46:29.019225 2987 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41947
I20260928 02:46:29.019429 2987 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:29.019965 2987 heartbeater.cc:511] Master 127.28.248.190:41947 requested a full tablet report, sending...
I20260928 02:46:29.021754 2796 ts_manager.cc:194] Registered new tserver with Master: 427a3da47f214912a622d16f38d5da07 (127.28.248.129:43579)
I20260928 02:46:29.023602 2796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:35531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:29.035683 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41947
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:29.147801 2992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:29.148077 2992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:29.148121 2992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:29.152549 2992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:29.152661 2992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:29.152738 2992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:29.158290 2992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:29.159375 2992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:29.160485 2992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:29.170651 2997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:29.170657 2998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:29.170771 2992 server_base.cc:1034] running on GCE node
W20260928 02:46:29.170657 3000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:29.171317 2992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:29.171962 2992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:29.173142 2992 hybrid_clock.cc:648] HybridClock initialized: now 1790563589173120 us; error 25 us; skew 500 ppm
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.175901 2992 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:29.176851 2992 webserver.cc:460] Webserver started at http://127.28.248.130:45911/ using document root <none> and password file <none>
I20260928 02:46:29.177419 2992 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:29.177481 2992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:29.177717 2992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:29.179700 2992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "a956385a0bd34c0eb1e19a92cf0e25f3"
format_stamp: "Formatted at 2026-09-28 02:46:29 on dist-test-slave-r238"
server_key: "f91d633f8e562f7545285f786f01d72b"
server_key_iv: "14b2809fc3127637e73a3baea4534294"
server_key_version: "encryptionkey@0"
I20260928 02:46:29.180260 2992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "a956385a0bd34c0eb1e19a92cf0e25f3"
format_stamp: "Formatted at 2026-09-28 02:46:29 on dist-test-slave-r238"
server_key: "f91d633f8e562f7545285f786f01d72b"
server_key_iv: "14b2809fc3127637e73a3baea4534294"
server_key_version: "encryptionkey@0"
I20260928 02:46:29.183992 2992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:46:29.186509 3007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:29.187776 2992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:29.187938 2992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "a956385a0bd34c0eb1e19a92cf0e25f3"
format_stamp: "Formatted at 2026-09-28 02:46:29 on dist-test-slave-r238"
server_key: "f91d633f8e562f7545285f786f01d72b"
server_key_iv: "14b2809fc3127637e73a3baea4534294"
server_key_version: "encryptionkey@0"
I20260928 02:46:29.188059 2992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:29.215420 2992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:29.218535 2992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:29.218743 2992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:29.219338 2992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:29.220187 2992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:29.220258 2992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:29.220328 2992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:29.220379 2992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:29.228820 2992 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:37467
I20260928 02:46:29.228845 3120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:37467 every 8 connection(s)
I20260928 02:46:29.229696 2992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:29.232391 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 2992
I20260928 02:46:29.232499 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:29.232803 29666 external_mini_cluster.cc:1428] Setting key d3374915a47c055f6f027552452bfd01
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:29.243824 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.231521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:38773 (local address 127.28.248.190:41947)
0928 02:46:29.231655 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.231659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.232487 (+ 828us) server_negotiation.cc:407] Connection header received
0928 02:46:29.234073 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.234076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.234129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.234236 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.235475 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.235994 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.237204 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.237434 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.237554 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.240243 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:29.240247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:29.240249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:29.240271 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:29.241743 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.242311 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.242315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.242317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.242397 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.242749 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.242754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.242756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.242935 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:29.243044 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:29.243534 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 02:46:29.243659 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:46:29.244705 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.231906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41947 (local address 127.28.248.130:38773)
0928 02:46:29.232339 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:29.232380 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:29.233860 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:29.234422 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:29.234433 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:29.234845 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:29.235327 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:29.235341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.236117 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.236127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:29.237082 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:29.237092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.237591 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.237595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:29.237766 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.238396 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:29.238412 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:29.240076 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:29.241861 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:29.241867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:29.241882 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:29.242208 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:29.242508 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:29.242511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:29.242513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:29.242653 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:29.243040 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:29.243048 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:29.243300 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:46:29.243493 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 02:46:29.243819 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:46:29.245935 3121 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41947
I20260928 02:46:29.246210 3121 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:29.246693 3121 heartbeater.cc:511] Master 127.28.248.190:41947 requested a full tablet report, sending...
I20260928 02:46:29.247763 2796 ts_manager.cc:194] Registered new tserver with Master: a956385a0bd34c0eb1e19a92cf0e25f3 (127.28.248.130:37467)
I20260928 02:46:29.248680 2796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:38773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:29.261183 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41947
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:29.372344 3126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:29.372686 3126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:29.372783 3126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:29.377080 3126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:29.377138 3126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:29.377216 3126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:29.382594 3126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:29.383677 3126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:29.384758 3126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:29.394688 3134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:29.394697 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:29.394688 3131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:29.394986 3126 server_base.cc:1034] running on GCE node
I20260928 02:46:29.395403 3126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:29.396114 3126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:29.397298 3126 hybrid_clock.cc:648] HybridClock initialized: now 1790563589397284 us; error 35 us; skew 500 ppm
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.399981 3126 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:29.400943 3126 webserver.cc:460] Webserver started at http://127.28.248.131:38291/ using document root <none> and password file <none>
I20260928 02:46:29.401525 3126 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:29.401618 3126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:29.401844 3126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:29.403884 3126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "22fbb9d8fac84b22aa75488d491970aa"
format_stamp: "Formatted at 2026-09-28 02:46:29 on dist-test-slave-r238"
server_key: "a1d386f2f9caeba95681482297edd4e7"
server_key_iv: "d107acf9782976892ae2079423c1ee0a"
server_key_version: "encryptionkey@0"
I20260928 02:46:29.404454 3126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "22fbb9d8fac84b22aa75488d491970aa"
format_stamp: "Formatted at 2026-09-28 02:46:29 on dist-test-slave-r238"
server_key: "a1d386f2f9caeba95681482297edd4e7"
server_key_iv: "d107acf9782976892ae2079423c1ee0a"
server_key_version: "encryptionkey@0"
I20260928 02:46:29.408273 3126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:29.410983 3141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:29.412431 3126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:29.412582 3126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "22fbb9d8fac84b22aa75488d491970aa"
format_stamp: "Formatted at 2026-09-28 02:46:29 on dist-test-slave-r238"
server_key: "a1d386f2f9caeba95681482297edd4e7"
server_key_iv: "d107acf9782976892ae2079423c1ee0a"
server_key_version: "encryptionkey@0"
I20260928 02:46:29.412757 3126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:29.441376 3126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:29.444330 3126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:29.444550 3126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:29.445185 3126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:29.446051 3126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:29.446123 3126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:29.446209 3126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:29.446259 3126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:29.455186 3126 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:46447
I20260928 02:46:29.455209 3254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:46447 every 8 connection(s)
I20260928 02:46:29.456131 3126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:29.457692 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3126
I20260928 02:46:29.457829 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:29.458139 29666 external_mini_cluster.cc:1428] Setting key 8bf9acd8d3e0c1837cab6208bdc7fecd
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:29.469877 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.458031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:58455 (local address 127.28.248.190:41947)
0928 02:46:29.458174 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.458178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.459180 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 02:46:29.460787 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.460790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.460847 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.460925 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.462139 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.462678 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.463814 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.464058 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.464169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.466597 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:29.466600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:29.466603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:29.466626 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:29.468090 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.468543 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.468545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.468546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.468667 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.468969 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.468971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.468972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.469139 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:29.469214 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:29.469662 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 02:46:29.469755 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:46:29.470763 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.458523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41947 (local address 127.28.248.131:58455)
0928 02:46:29.459032 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:29.459085 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:29.460530 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:29.461097 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:29.461104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:29.461521 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:29.462002 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:29.462016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.462816 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.462818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:29.463688 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:29.463697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.464206 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.464210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:29.464372 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.464969 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:29.464985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:29.466439 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:29.468193 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:29.468197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:29.468209 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:29.468455 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:29.468745 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:29.468747 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:29.468748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:29.468880 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:29.469224 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:29.469229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:29.469458 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 02:46:29.469621 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:46:29.469868 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":141,"threads_started":1}
I20260928 02:46:29.471956 3255 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41947
I20260928 02:46:29.472240 3255 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:29.472841 3255 heartbeater.cc:511] Master 127.28.248.190:41947 requested a full tablet report, sending...
I20260928 02:46:29.473896 2796 ts_manager.cc:194] Registered new tserver with Master: 22fbb9d8fac84b22aa75488d491970aa (127.28.248.131:46447)
I20260928 02:46:29.474424 2796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:58455
I20260928 02:46:29.484145 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/test-admin.keytab.
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.501401 29666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:46:29.504151 3262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:29.502792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.28.248.129:43579)
0928 02:46:29.503097 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.503112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.503135 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:46:29.503184 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.503189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.503374 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.503497 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.503968 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60920: BlockingRecv error: recv got EOF from 127.0.0.1:60920 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
W20260928 02:46:29.504590 29666 init.cc:244] Error while opening credential cache '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc)
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.505640 29666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.506556 29666 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:29.515839 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.507414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.28.248.129:43579)
0928 02:46:29.507565 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.507570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.507651 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:46:29.507746 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.507749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.507786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.507851 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.508336 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.509187 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.509974 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.510384 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.510471 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.512756 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:29.512764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:29.512777 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:29.512798 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:29.514258 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.514638 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.514643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.514644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.514718 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.514935 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.514937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.514938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.515152 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:29.515311 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:29.515503 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:46:29.515673 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:46:29.519924 3262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:29.518897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.28.248.129:43579)
0928 02:46:29.519025 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.519029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.519137 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:46:29.519278 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.519281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.519318 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.519407 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.519793 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60938: BlockingRecv error: recv got EOF from 127.0.0.1:60938 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:46:29.520076 29666 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.521131 29666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.522015 29666 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:29.530085 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.522800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.28.248.129:43579)
0928 02:46:29.522920 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.522924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.523034 (+ 110us) server_negotiation.cc:407] Connection header received
0928 02:46:29.523174 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.523177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.523219 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.523300 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.523774 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.524285 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.525124 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.525360 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.525473 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.527602 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:29.527605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:29.527607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:29.527627 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:29.528849 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.529203 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.529205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.529206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.529272 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.529484 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.529486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.529487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.529625 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:29.529716 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:29.529894 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:46:29.529980 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 02:46:29.532840 3262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:29.531824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.28.248.129:43579)
0928 02:46:29.531932 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.531936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.532124 (+ 188us) server_negotiation.cc:407] Connection header received
0928 02:46:29.532215 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.532217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.532249 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.532312 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.532731 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60948: BlockingRecv error: recv got EOF from 127.0.0.1:60948 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:46:29.532976 29666 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563560542449-29666-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.533964 29666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:29.534873 29666 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[2751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:46:29.543778 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.535698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.28.248.129:43579)
0928 02:46:29.535825 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:29.535828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:29.535840 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:29.535969 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:29.535972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:29.536005 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:29.536067 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:29.536510 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.537482 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.538250 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:29.538471 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:29.538555 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:29.540964 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:29.540967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:29.540969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:29.540992 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:29.542340 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.542724 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.542726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.542727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.542799 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:29.543029 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:29.543032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:29.543033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:29.543184 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:29.543278 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:29.543464 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:46:29.543552 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:46:29.544891 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2858
I20260928 02:46:29.550551 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2992
I20260928 02:46:29.556051 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3126
I20260928 02:46:29.561285 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 2764
2026-09-28T02:46:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2219 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[3275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[3275](info): set up 2 sockets
Sep 28 02:46:29 dist-test-slave-r238 krb5kdc[3275](info): commencing operation
krb5kdc: starting...
W20260928 02:46:30.609311 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:30 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab.
Entry for principal oryx/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab.
I20260928 02:46:30.736276 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43167
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab
--principal=oryx/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:30.850325 3288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:30.850660 3288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:30.850705 3288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:30.855029 3288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:30.855113 3288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:30.855134 3288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:30.855150 3288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:30.855196 3288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:30.860966 3288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:43167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab
--principal=oryx/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:30.862161 3288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:30.863387 3288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:30.872822 3294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:30.872845 3293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:30.872915 3288 server_base.cc:1034] running on GCE node
W20260928 02:46:30.873000 3296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:30.873688 3288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:30.875016 3288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:30.876183 3288 hybrid_clock.cc:648] HybridClock initialized: now 1790563590876169 us; error 28 us; skew 500 ppm
Sep 28 02:46:30 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:30.879050 3288 init.cc:377] Logged in from keytab as oryx/127.28.248.190@KRBTEST.COM (short username oryx)
I20260928 02:46:30.879920 3288 webserver.cc:460] Webserver started at http://127.28.248.190:38407/ using document root <none> and password file <none>
I20260928 02:46:30.880436 3288 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:30.880503 3288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:30.880738 3288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:30.882848 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "ab9e3d501cae4f29af1d1f6aaeb76970"
format_stamp: "Formatted at 2026-09-28 02:46:30 on dist-test-slave-r238"
server_key: "547b0a9befddabc9b476b9d1f7651c5b"
server_key_iv: "e5862ecaae2e933ec93a59444e2e5aaa"
server_key_version: "encryptionkey@0"
I20260928 02:46:30.883363 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "ab9e3d501cae4f29af1d1f6aaeb76970"
format_stamp: "Formatted at 2026-09-28 02:46:30 on dist-test-slave-r238"
server_key: "547b0a9befddabc9b476b9d1f7651c5b"
server_key_iv: "e5862ecaae2e933ec93a59444e2e5aaa"
server_key_version: "encryptionkey@0"
I20260928 02:46:30.887187 3288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:46:30.889662 3303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:30.891089 3288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:30.891258 3288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "ab9e3d501cae4f29af1d1f6aaeb76970"
format_stamp: "Formatted at 2026-09-28 02:46:30 on dist-test-slave-r238"
server_key: "547b0a9befddabc9b476b9d1f7651c5b"
server_key_iv: "e5862ecaae2e933ec93a59444e2e5aaa"
server_key_version: "encryptionkey@0"
I20260928 02:46:30.891386 3288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:30.911537 3288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:30.915181 3288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:30.915422 3288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:30.923128 3288 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:43167
I20260928 02:46:30.923401 3355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:43167 every 8 connection(s)
I20260928 02:46:30.924257 3288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:30.927169 3356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:30.932897 3356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Bootstrap starting.
I20260928 02:46:30.933420 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3288
I20260928 02:46:30.933586 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:30.933872 29666 external_mini_cluster.cc:1428] Setting key 7e5120b1c5f781e39e5c93fbdd4f3671
I20260928 02:46:30.935803 3356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:30.936555 3356 log.cc:824] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:30.938499 3356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: No bootstrap required, opened a new log
I20260928 02:46:30.942081 3356 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } }
I20260928 02:46:30.942354 3356 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:30.942425 3356 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9e3d501cae4f29af1d1f6aaeb76970, State: Initialized, Role: FOLLOWER
Sep 28 02:46:30 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:30.942890 3356 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } }
I20260928 02:46:30.943076 3356 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:30.943158 3356 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:30.943272 3356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:30.944594 3356 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } }
I20260928 02:46:30.945040 3356 leader_election.cc:302] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9e3d501cae4f29af1d1f6aaeb76970; no voters:
I20260928 02:46:30.945407 3356 leader_election.cc:288] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:30.945542 3361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:30.945798 3361 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 LEADER]: Becoming Leader. State: Replica: ab9e3d501cae4f29af1d1f6aaeb76970, State: Running, Role: LEADER
I20260928 02:46:30.946187 3361 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } }
I20260928 02:46:30.946898 3356 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:30.948781 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } }
I20260928 02:46:30.948927 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:30.949350 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:30.935229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.28.248.190:43167)
0928 02:46:30.935746 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:30.935761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:30.935798 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:46:30.936532 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:30.936551 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:30.937012 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:30.937358 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:30.938160 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:30.939206 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:30.940071 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:30.940580 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:30.940824 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:30.943505 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:30.943515 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:30.943532 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:30.943569 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:30.945660 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:30.946210 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:30.946216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:30.946221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:30.946356 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:30.946704 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:30.946707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:30.946709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:30.947268 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:30.947471 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:30.947856 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:46:30.948199 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":175,"threads_started":1}
I20260928 02:46:30.949689 3363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9e3d501cae4f29af1d1f6aaeb76970. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } }
I20260928 02:46:30.949803 3363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:30.951809 3375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:30.951973 3375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:30.952087 3376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:30.952936 3376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:30.959019 3376 catalog_manager.cc:1347] Generated new cluster ID: 078e495dc1a7446c98c0039f003aeff4
I20260928 02:46:30.959096 3376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:30.967084 3376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:30.968051 3376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:30.979277 3376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Generated new TSK 0
I20260928 02:46:30.980208 3376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab.
Entry for principal oryx/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab.
I20260928 02:46:31.015614 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43167
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:31.134697 3382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:31.135025 3382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:31.135070 3382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:31.139267 3382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:31.139344 3382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:31.139406 3382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:31.144870 3382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:31.146090 3382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:31.147272 3382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:31.157424 3388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:31.157392 3387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:31.157564 3382 server_base.cc:1034] running on GCE node
W20260928 02:46:31.157853 3390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:31.158336 3382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:31.159026 3382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:31.160221 3382 hybrid_clock.cc:648] HybridClock initialized: now 1790563591160207 us; error 26 us; skew 500 ppm
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:31.163300 3382 init.cc:377] Logged in from keytab as oryx/127.28.248.129@KRBTEST.COM (short username oryx)
I20260928 02:46:31.164207 3382 webserver.cc:460] Webserver started at http://127.28.248.129:36451/ using document root <none> and password file <none>
I20260928 02:46:31.164860 3382 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:31.164953 3382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:31.165191 3382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:31.167225 3382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "e05df32e19e040589e50754445d84c74"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "ded701b5bfcfb65c0bc5ef1ae055654a"
server_key_iv: "0549f5f824e1236cdc3f5e00efffdbed"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.167799 3382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "e05df32e19e040589e50754445d84c74"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "ded701b5bfcfb65c0bc5ef1ae055654a"
server_key_iv: "0549f5f824e1236cdc3f5e00efffdbed"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.172220 3382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:46:31.175364 3397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.176995 3382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:31.177170 3382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "e05df32e19e040589e50754445d84c74"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "ded701b5bfcfb65c0bc5ef1ae055654a"
server_key_iv: "0549f5f824e1236cdc3f5e00efffdbed"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.177313 3382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:31.196381 3382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:31.200068 3382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:31.200307 3382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:31.201004 3382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:31.201915 3382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:31.201998 3382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.202075 3382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:31.202121 3382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.211660 3382 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:42913
I20260928 02:46:31.211678 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:42913 every 8 connection(s)
I20260928 02:46:31.212770 3382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:31.213603 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3382
I20260928 02:46:31.213721 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:31.214021 29666 external_mini_cluster.cc:1428] Setting key f4fd2b9f95e59c7621efc530ca7f4f60
WARNING: no policy specified for oryx/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.130@KRBTEST.COM" created.
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:31.234308 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.214826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:38321 (local address 127.28.248.190:43167)
0928 02:46:31.214962 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.214971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.216064 (+ 1093us) server_negotiation.cc:407] Connection header received
0928 02:46:31.217582 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.217588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.217649 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.217745 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.218986 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.219915 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.221948 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.222295 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.222432 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.229492 (+ 7060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.229496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.229498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.229531 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:31.231366 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.232154 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.232163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.232164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.232255 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.232785 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.232792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.232796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.233104 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:31.233224 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:31.233856 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 02:46:31.234053 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:46:31.235131 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.215380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:38321)
0928 02:46:31.215882 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.215947 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.217360 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.217909 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.217916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.218330 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.218834 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.218847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.220080 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.220087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.221782 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.221795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.222524 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.222531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.222791 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.223505 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.223533 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.229192 (+ 5659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:31.231524 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.231531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.231547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.231904 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.232371 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.232375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.232378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.232565 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.233230 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:31.233237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:31.233553 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 02:46:31.233732 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:46:31.234067 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":141,"threads_started":1}
I20260928 02:46:31.236795 3511 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43167
I20260928 02:46:31.237118 3511 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:31.237732 3511 heartbeater.cc:511] Master 127.28.248.190:43167 requested a full tablet report, sending...
I20260928 02:46:31.239745 3320 ts_manager.cc:194] Registered new tserver with Master: e05df32e19e040589e50754445d84c74 (127.28.248.129:42913)
I20260928 02:46:31.241111 3320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:38321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab.
Entry for principal oryx/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab.
I20260928 02:46:31.247720 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43167
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:31.360504 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:31.360880 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:31.360972 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:31.365118 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:31.365221 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:31.365324 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:31.370888 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:31.372166 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:31.373312 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:31.383838 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:31.384063 3516 server_base.cc:1034] running on GCE node
W20260928 02:46:31.383841 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:31.384223 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:31.384769 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:31.385519 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:31.386718 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790563591386700 us; error 41 us; skew 500 ppm
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:31.389772 3516 init.cc:377] Logged in from keytab as oryx/127.28.248.130@KRBTEST.COM (short username oryx)
I20260928 02:46:31.390776 3516 webserver.cc:460] Webserver started at http://127.28.248.130:33597/ using document root <none> and password file <none>
I20260928 02:46:31.391378 3516 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:31.391610 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:31.391887 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:31.394126 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "4f4e91168a294f54942159198b96ef85"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "9ff41d246d89c1dcf4d4fb267b982fe5"
server_key_iv: "939ed619b33c51ca29c503eb9ee08dca"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.394768 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "4f4e91168a294f54942159198b96ef85"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "9ff41d246d89c1dcf4d4fb267b982fe5"
server_key_iv: "939ed619b33c51ca29c503eb9ee08dca"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.398891 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:31.401585 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.402897 3516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:31.403054 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "4f4e91168a294f54942159198b96ef85"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "9ff41d246d89c1dcf4d4fb267b982fe5"
server_key_iv: "939ed619b33c51ca29c503eb9ee08dca"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.403179 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:31.428098 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:31.431195 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:31.431422 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:31.432076 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:31.433001 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:31.433073 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.433147 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:31.433200 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.445546 3516 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:40821
I20260928 02:46:31.445567 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:40821 every 8 connection(s)
I20260928 02:46:31.446722 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:31.455227 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3516
I20260928 02:46:31.455427 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:31.455850 29666 external_mini_cluster.cc:1428] Setting key b5de370e47a3ebf6defed10c51b205cf
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:31.466086 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.449228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:48585 (local address 127.28.248.190:43167)
0928 02:46:31.449432 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.449437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.450841 (+ 1404us) server_negotiation.cc:407] Connection header received
0928 02:46:31.453012 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.453017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.453084 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.453177 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.455000 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.455858 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.457311 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.457627 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.457759 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.461344 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.461351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.461357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.461393 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:31.463438 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.464168 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.464174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.464179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.464283 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.464723 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.464729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.464733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.465023 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:31.465135 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:31.465725 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 02:46:31.465878 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:31.466846 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.449930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:48585)
0928 02:46:31.450568 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.450633 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.452737 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.453411 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.453418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.454022 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.454779 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.454806 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.456052 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.456055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.457177 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.457187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.457806 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.457810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.458079 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.458842 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.458869 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.461111 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:31.463578 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.463584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.463598 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.464030 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.464388 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.464391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.464392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.464554 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.465145 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:31.465151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:31.465443 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:46:31.465617 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:46:31.465931 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for oryx/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:31.468206 3645 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43167
I20260928 02:46:31.468505 3645 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:31.469154 3645 heartbeater.cc:511] Master 127.28.248.190:43167 requested a full tablet report, sending...
I20260928 02:46:31.470374 3320 ts_manager.cc:194] Registered new tserver with Master: 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821)
I20260928 02:46:31.471275 3320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:48585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab.
Entry for principal oryx/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab.
I20260928 02:46:31.491168 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43167
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:31.604239 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:31.604606 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:31.604727 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:31.609071 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:31.609172 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:31.609272 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:31.614883 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:31.616135 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:31.617288 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:31.627431 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:31.627431 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:31.627427 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:31.627637 3650 server_base.cc:1034] running on GCE node
I20260928 02:46:31.628258 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:31.629014 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:31.630198 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790563591630181 us; error 31 us; skew 500 ppm
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:31.633342 3650 init.cc:377] Logged in from keytab as oryx/127.28.248.131@KRBTEST.COM (short username oryx)
I20260928 02:46:31.634265 3650 webserver.cc:460] Webserver started at http://127.28.248.131:43291/ using document root <none> and password file <none>
I20260928 02:46:31.634833 3650 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:31.634894 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:31.635069 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:31.637499 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "d3861405cbecf5e8f9ed105731162689"
server_key_iv: "c4ae635ad29357e3409223244bbf3817"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.638046 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "d3861405cbecf5e8f9ed105731162689"
server_key_iv: "c4ae635ad29357e3409223244bbf3817"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.642447 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:46:31.644986 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.646276 3650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:31.646407 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "d3861405cbecf5e8f9ed105731162689"
server_key_iv: "c4ae635ad29357e3409223244bbf3817"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.646557 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:31.665907 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:31.669227 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:31.669435 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:31.670096 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:31.670969 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:31.671020 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.671069 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:31.671092 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.680339 3650 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:38611
I20260928 02:46:31.680353 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:38611 every 8 connection(s)
I20260928 02:46:31.681388 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:31.688287 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3650
I20260928 02:46:31.688431 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:31.688791 29666 external_mini_cluster.cc:1428] Setting key f9ac3e2fe1c6dfc2d3c73a7d1b3c0ca3
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:31.695712 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.683423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:51971 (local address 127.28.248.190:43167)
0928 02:46:31.683588 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.683593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.684440 (+ 847us) server_negotiation.cc:407] Connection header received
0928 02:46:31.685912 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.685915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.685987 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.686099 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.687365 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.687904 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.689337 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.689640 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.689744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.692454 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.692457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.692459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.692486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:31.693845 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.694337 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.694340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.694341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.694416 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.694714 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.694716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.694718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.694932 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:31.695049 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:31.695480 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 02:46:31.695586 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:46:31.696712 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.683778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:51971)
0928 02:46:31.684268 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.684314 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.685675 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.686286 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.686293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.686718 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.687213 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.687228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.688042 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.688050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.689185 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.689197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.689804 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.689810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.689994 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.690682 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.690699 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.692292 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:31.693953 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.693959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.693974 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.694244 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.694502 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.694504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.694505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.694639 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.695043 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:31.695048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:31.695291 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 02:46:31.695451 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 02:46:31.695732 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
I20260928 02:46:31.697903 3779 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43167
I20260928 02:46:31.698160 3779 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:31.698673 3779 heartbeater.cc:511] Master 127.28.248.190:43167 requested a full tablet report, sending...
I20260928 02:46:31.699717 3320 ts_manager.cc:194] Registered new tserver with Master: c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611)
I20260928 02:46:31.700311 3320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:51971
I20260928 02:46:31.702778 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.248.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.189.keytab.
Entry for principal oryx/127.28.248.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.189.keytab.
I20260928 02:46:31.731557 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:43751
--webserver_interface=127.28.248.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.189.keytab
--principal=oryx/127.28.248.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:43167,127.28.248.189:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:31.847643 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:31.848054 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:31.848109 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:31.852517 3784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:31.852650 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:31.852690 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:31.852732 3784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:31.852756 3784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:31.858381 3784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:43167,127.28.248.189:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.189.keytab
--principal=oryx/127.28.248.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.248.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
W20260928 02:46:31.858615 3784 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.248.190:43167,127.28.248.189:43751'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:46:31.859525 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:31.860533 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:31.870357 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:31.870362 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:31.870635 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:31.870805 3784 server_base.cc:1034] running on GCE node
I20260928 02:46:31.871397 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:31.872592 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:31.873813 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790563591873796 us; error 30 us; skew 500 ppm
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:31.876559 3784 init.cc:377] Logged in from keytab as oryx/127.28.248.189@KRBTEST.COM (short username oryx)
I20260928 02:46:31.877530 3784 webserver.cc:460] Webserver started at http://127.28.248.189:45679/ using document root <none> and password file <none>
I20260928 02:46:31.878105 3784 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:31.878197 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:31.878424 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:31.880539 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/instance:
uuid: "c32337296f7d48c0bc020e48af6629b5"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "64bc700a15c96334033e99b4f5ac899b"
server_key_iv: "121ca07eeeeb0ec5d7aa4b1663d66600"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.881268 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal/instance:
uuid: "c32337296f7d48c0bc020e48af6629b5"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "64bc700a15c96334033e99b4f5ac899b"
server_key_iv: "121ca07eeeeb0ec5d7aa4b1663d66600"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.885244 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:46:31.887895 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:31.889266 3784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:31.889413 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
uuid: "c32337296f7d48c0bc020e48af6629b5"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "64bc700a15c96334033e99b4f5ac899b"
server_key_iv: "121ca07eeeeb0ec5d7aa4b1663d66600"
server_key_version: "encryptionkey@0"
I20260928 02:46:31.889539 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:31.911612 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:31.915055 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:31.915266 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:31.922819 3784 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.189:43751
I20260928 02:46:31.923197 3851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.189:43751 every 8 connection(s)
I20260928 02:46:31.924064 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
I20260928 02:46:31.926960 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:31.928574 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3784
I20260928 02:46:31.931404 3852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.189@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:31.944741 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.933788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.28.248.190:43167)
0928 02:46:31.933940 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.933945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.934312 (+ 367us) server_negotiation.cc:407] Connection header received
0928 02:46:31.934546 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.934549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.934601 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.934690 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.935416 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.935928 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.937187 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.937565 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.937697 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.940409 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.940413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.940416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.940450 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:31.942146 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.943139 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.943143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.943145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.943230 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.943511 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.943514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.943516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.943870 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:31.943980 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:31.944390 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 02:46:31.944519 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:46:31.945597 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.930265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.28.248.189:43751)
0928 02:46:31.930754 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.930773 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.930820 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:46:31.931721 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.931745 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.932174 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.932525 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.933446 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.934908 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.936089 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.936824 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.936973 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.939846 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.939857 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.939873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.939898 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:31.942132 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.942597 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.942602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.942607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.942734 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.943060 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.943063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.943064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.943730 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:31.943986 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:31.944213 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:46:31.944566 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260928 02:46:31.945597 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.933756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:45014)
0928 02:46:31.934197 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.934244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.934395 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.934746 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.934751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.935173 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.935284 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.935289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.936065 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.936070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.937074 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.937082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.937737 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.937740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.937846 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.938433 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.938452 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.940191 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:31.942602 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.942610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.942614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.943031 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.943330 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.943332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.943333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.943412 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.943990 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:31.943997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:31.944159 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 02:46:31.944310 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:46:31.944675 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":14,"thread_start_us":159,"threads_started":1}
I20260928 02:46:31.947006 3852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:31 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563591, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.189@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
I20260928 02:46:31.955931 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.947642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:50404)
0928 02:46:31.947802 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.947816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.947905 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.948231 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.948234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.948425 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.948513 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.948519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.949675 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.949681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.950275 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.950283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.950839 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.950842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.950951 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.951412 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.951424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.953038 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:31.954610 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.954613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.954615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.954809 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.955061 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:31.955063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:31.955064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:31.955140 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:31.955499 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:31.955500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:31.955576 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 02:46:31.955653 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:46:31.955757 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 02:46:31.955950 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.947820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.28.248.189:43751)
0928 02:46:31.947953 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.947958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.947975 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:46:31.948016 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.948019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.948066 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.948223 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.948663 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.949549 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.950403 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.950716 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.950832 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.953172 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.953176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.953179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.953207 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:31.954502 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.954904 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.954906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.954907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.954979 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:31.955219 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:31.955221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:31.955222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:31.955415 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:31.955490 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:31.955708 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:46:31.955821 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:46:31.959290 3852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Bootstrap starting.
I20260928 02:46:31.962090 3852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:31.962893 3852 log.cc:824] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:31.965145 3852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: No bootstrap required, opened a new log
I20260928 02:46:31.968278 3852 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } }
I20260928 02:46:31.968505 3852 raft_consensus.cc:374] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:31.968581 3852 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32337296f7d48c0bc020e48af6629b5, State: Initialized, Role: FOLLOWER
I20260928 02:46:31.969069 3852 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } }
I20260928 02:46:31.969539 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } } }
I20260928 02:46:31.969661 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:31.970294 3852 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:46:31.976361 3869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:31.976464 3869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:31.978566 3784 master_runner.cc:428] Detected that this master c32337296f7d48c0bc020e48af6629b5 is joining an existing cluster
I20260928 02:46:31.978674 3784 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.248.189:43751
I20260928 02:46:32.003525 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.991030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:50408)
0928 02:46:31.991392 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.991409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.991533 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.993006 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.993009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.993310 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.993472 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.993482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.995401 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.995406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.996353 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.996363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.997046 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.997049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.997156 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.997768 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.997788 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.998173 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.000341 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.000349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.000353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.000657 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.001296 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.001304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.001307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.001394 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.002860 (+ 1466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.002864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.002962 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.003078 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.003259 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260928 02:46:32.004158 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.991740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.28.248.190:43167)
0928 02:46:31.991878 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.991885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.992355 (+ 470us) server_negotiation.cc:407] Connection header received
0928 02:46:31.992490 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.992495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.992555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.992655 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.994510 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.995271 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.996638 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.996970 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.997048 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.999296 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.999303 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.999309 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.999345 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.002308 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.002799 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.002803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.002805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.002909 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.003233 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.003237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.003239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.003467 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.003542 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.003882 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.004008 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:32.004669 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.992696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.28.248.189:43751)
0928 02:46:31.992792 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:31.992796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:31.992810 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:31.992851 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:31.992854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:31.992897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:31.992994 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:31.993592 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.994439 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.996480 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.996808 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.996894 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.998327 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:31.998331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:31.998334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:31.998358 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.000244 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.001104 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.001108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.001112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.001204 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.002439 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.002442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.002444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.002729 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.002823 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.004096 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.004456 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:46:32.006294 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.991982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:45030)
0928 02:46:31.992271 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.992285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:31.992390 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:31.992810 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:31.992814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:31.994239 (+ 1425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:31.994365 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.994372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.995502 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.995507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.996503 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:31.996510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:31.997196 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:31.997199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:31.997300 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:31.998587 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:31.998606 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:31.999119 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.002424 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.002429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.002431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.002677 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.003029 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.003033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.003035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.003126 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.003611 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.003614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.003704 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.004301 (+ 597us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.006126 (+ 1825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1715,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for oryx/127.28.248.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.188@KRBTEST.COM" created.
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.190@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
I20260928 02:46:32.026902 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.012201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:50418)
0928 02:46:32.012600 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.012680 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.012865 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.013306 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.013312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.013673 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.013831 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.013866 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.014922 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.014927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.016358 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.016381 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.017296 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.017300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.017442 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.019707 (+ 2265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.019728 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.021984 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.024535 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.024550 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.024558 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.024968 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.025353 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.025359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.025361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.025463 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.026047 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.026058 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.026231 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.026399 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.026643 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
I20260928 02:46:32.027810 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.012263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.28.248.189:43751)
0928 02:46:32.012411 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.012416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.012756 (+ 340us) server_negotiation.cc:407] Connection header received
0928 02:46:32.013034 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.013038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.013098 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.013143 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.013995 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.014782 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.016536 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.016987 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.017086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.022211 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.022217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.022222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.022258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.024362 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.025130 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.025135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.025139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.025238 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.025581 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.025584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.025586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.025856 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.025959 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.027455 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.027611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:46:32.030189 3320 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.248.189:43751: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c32337296f7d48c0bc020e48af6629b5"
member_type: NON_VOTER
last_known_addr {
host: "127.28.248.189"
port: 43751
}
attrs {
promote: true
}
}
dest_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970"
cas_config_opid_index: -1
I20260928 02:46:32.030983 3320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } }
I20260928 02:46:32.033010 3858 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:32.033192 3858 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } }
I20260928 02:46:32.033936 3826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:32.034552 3858 leader_election.cc:288] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167)
I20260928 02:46:32.035003 3330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c32337296f7d48c0bc020e48af6629b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" is_pre_election: true
I20260928 02:46:32.034941 3882 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c32337296f7d48c0bc020e48af6629b5 (127.28.248.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.035413 3826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Refusing update from remote peer ab9e3d501cae4f29af1d1f6aaeb76970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:46:32.035815 3803 leader_election.cc:302] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c32337296f7d48c0bc020e48af6629b5; no voters: ab9e3d501cae4f29af1d1f6aaeb76970
I20260928 02:46:32.036044 3881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:32.036332 3858 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:46:32.037498 3882 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.248.189:43751
I20260928 02:46:32.038156 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.038261 3784 master.cc:537] Master@127.28.248.189:43751 shutting down...
I20260928 02:46:32.038444 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.188.keytab.
Entry for principal oryx/127.28.248.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.188.keytab.
I20260928 02:46:32.040707 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:45851
--webserver_interface=127.28.248.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.188.keytab
--principal=oryx/127.28.248.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:32.044389 3826 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c32337296f7d48c0bc020e48af6629b5 (127.28.248.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.046659 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9e3d501cae4f29af1d1f6aaeb76970. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.046787 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.046882 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.046948 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.047008 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.047068 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.048942 3784 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Raft consensus shutting down.
I20260928 02:46:32.049352 3784 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Raft consensus is shut down!
I20260928 02:46:32.049496 3784 tablet_replica.cc:330] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: stopping tablet replica
I20260928 02:46:32.077674 3784 master.cc:552] Master@127.28.248.189:43751 shutdown complete.
I20260928 02:46:32.077752 3784 master_runner.cc:305] Clearing existing system tablet
I20260928 02:46:32.077996 3784 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.078070 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.079386 3784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:46:32.080346 3894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.080583 3784 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.080768 3784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:32.080847 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
uuid: "c32337296f7d48c0bc020e48af6629b5"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "64bc700a15c96334033e99b4f5ac899b"
server_key_iv: "121ca07eeeeb0ec5d7aa4b1663d66600"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.082645 3784 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:32.084650 3784 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:46:32.084756 3784 log.cc:1196] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:46:32.085052 3784 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Deleting consensus metadata
I20260928 02:46:32.085237 3784 master_runner.cc:315] Copying system tablet from 127.28.248.190:43167
I20260928 02:46:32.086400 3784 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: tablet copy: Beginning tablet copy session from remote peer at address 127.28.248.190:43167
I20260928 02:46:32.094601 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.087072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.28.248.190:43167)
0928 02:46:32.087211 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.087215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.087440 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:46:32.087617 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.087620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.087676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.087766 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.088215 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.088870 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.089705 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.089938 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.090054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.091224 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.091228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.091231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.091257 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.092872 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.093331 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.093334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.093336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.093421 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.093711 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.093716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.093718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.093917 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.094037 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.094308 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.094428 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:32.094624 3900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.086997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:45036)
0928 02:46:32.087338 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.087351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.087497 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.087781 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.087784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.087996 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.088092 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.088098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.088976 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.088981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.089575 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.089584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.090058 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.090061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.090145 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.090738 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.090751 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.091083 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.093003 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.093008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.093011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.093226 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.093526 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.093529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.093531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.093622 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.094025 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.094028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.094138 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.094243 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.094414 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
I20260928 02:46:32.095562 3340 tablet_copy_service.cc:140] P ab9e3d501cae4f29af1d1f6aaeb76970: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c32337296f7d48c0bc020e48af6629b5 ({username='oryx', principal='oryx/127.28.248.189@KRBTEST.COM'} at 127.0.0.1:45036)
I20260928 02:46:32.095722 3340 tablet_copy_service.cc:161] P ab9e3d501cae4f29af1d1f6aaeb76970: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c32337296f7d48c0bc020e48af6629b5 at {username='oryx', principal='oryx/127.28.248.189@KRBTEST.COM'} at 127.0.0.1:45036: session id = c32337296f7d48c0bc020e48af6629b5-00000000000000000000000000000000
I20260928 02:46:32.097364 3340 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:46:32.098881 3784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:46:32.100885 3784 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: tablet copy: Starting download of 0 data blocks...
I20260928 02:46:32.101095 3784 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: tablet copy: Starting download of 1 WAL segments...
I20260928 02:46:32.102816 3784 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:46:32.104440 3340 tablet_copy_service.cc:342] P ab9e3d501cae4f29af1d1f6aaeb76970: Request end of tablet copy session c32337296f7d48c0bc020e48af6629b5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.248.189@KRBTEST.COM'} at 127.0.0.1:45036
I20260928 02:46:32.104561 3340 tablet_copy_service.cc:433] P ab9e3d501cae4f29af1d1f6aaeb76970: ending tablet copy session c32337296f7d48c0bc020e48af6629b5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c32337296f7d48c0bc020e48af6629b5
I20260928 02:46:32.111514 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:32.113368 3901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.113370 3902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.113828 3784 server_base.cc:1034] running on GCE node
W20260928 02:46:32.116765 3904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.117025 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:32.117380 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:32.118520 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790563592118530 us; error 45 us; skew 500 ppm
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:32.120062 3784 init.cc:377] Logged in from keytab as oryx/127.28.248.189@KRBTEST.COM (short username oryx)
I20260928 02:46:32.120575 3784 webserver.cc:460] Webserver started at http://127.28.248.189:45257/ using document root <none> and password file <none>
I20260928 02:46:32.120813 3784 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.120872 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.122030 3784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:32.123030 3911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.123260 3784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:32.123354 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
uuid: "c32337296f7d48c0bc020e48af6629b5"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "64bc700a15c96334033e99b4f5ac899b"
server_key_iv: "121ca07eeeeb0ec5d7aa4b1663d66600"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.123461 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.147750 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:32.150149 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:32.155298 3784 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.189:43751
I20260928 02:46:32.155310 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.189:43751 every 8 connection(s)
I20260928 02:46:32.155884 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
I20260928 02:46:32.157689 3964 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:32.158546 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Bootstrap starting.
I20260928 02:46:32.165957 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:32.166304 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: Bootstrap complete.
I20260928 02:46:32.166846 3964 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } }
I20260928 02:46:32.166981 3964 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c32337296f7d48c0bc020e48af6629b5, State: Initialized, Role: LEARNER
I20260928 02:46:32.167109 3964 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } }
I20260928 02:46:32.167471 3967 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.167636 3967 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.167640 3964 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:32.171379 3978 catalog_manager.cc:1259] Loaded cluster ID: 078e495dc1a7446c98c0039f003aeff4
I20260928 02:46:32.171464 3978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: loading cluster ID for follower catalog manager: success
I20260928 02:46:32.173216 3978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: acquiring CA information for follower catalog manager: success
I20260928 02:46:32.174484 3978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:46:32.210671 3886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:32.210955 3886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:32.210996 3886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:32.215250 3886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:32.215308 3886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:32.215328 3886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:32.215345 3886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:32.215360 3886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:32.220538 3886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.188.keytab
--principal=oryx/127.28.248.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:45851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.248.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:32.221552 3886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:32.222491 3886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:32.231194 3983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.231194 3984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.231245 3986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.231493 3886 server_base.cc:1034] running on GCE node
I20260928 02:46:32.232111 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:32.233291 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:32.234452 3886 hybrid_clock.cc:648] HybridClock initialized: now 1790563592234441 us; error 30 us; skew 500 ppm
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:32.237272 3886 init.cc:377] Logged in from keytab as oryx/127.28.248.188@KRBTEST.COM (short username oryx)
I20260928 02:46:32.238237 3886 webserver.cc:460] Webserver started at http://127.28.248.188:32847/ using document root <none> and password file <none>
I20260928 02:46:32.238831 3886 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.238974 3886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.239216 3886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:32.241397 3886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/instance:
uuid: "a9f8fee6b4de43fc900574449d5cf91d"
format_stamp: "Formatted at 2026-09-28 02:46:32 on dist-test-slave-r238"
server_key: "c98693c80a060fccb96351549de6fa5e"
server_key_iv: "a0bf3294d3fd627a59a5b0e31a791d9f"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.241979 3886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal/instance:
uuid: "a9f8fee6b4de43fc900574449d5cf91d"
format_stamp: "Formatted at 2026-09-28 02:46:32 on dist-test-slave-r238"
server_key: "c98693c80a060fccb96351549de6fa5e"
server_key_iv: "a0bf3294d3fd627a59a5b0e31a791d9f"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.244029 3511 heartbeater.cc:503] Master 127.28.248.190:43167 was elected leader, sending a full tablet report...
I20260928 02:46:32.246126 3886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:46:32.248852 3993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.250247 3886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:32.250430 3886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
uuid: "a9f8fee6b4de43fc900574449d5cf91d"
format_stamp: "Formatted at 2026-09-28 02:46:32 on dist-test-slave-r238"
server_key: "c98693c80a060fccb96351549de6fa5e"
server_key_iv: "a0bf3294d3fd627a59a5b0e31a791d9f"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.250561 3886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.266137 3886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:32.270000 3886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:32.270210 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:32.277284 3886 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.188:45851
I20260928 02:46:32.277586 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.188:45851 every 8 connection(s)
I20260928 02:46:32.278278 3886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
I20260928 02:46:32.281106 4046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:32.284874 4046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:32.285632 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 3886
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.188@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.248.188@KRBTEST.COM
I20260928 02:46:32.299566 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.287136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.28.248.190:43167)
0928 02:46:32.287285 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.287290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.287653 (+ 363us) server_negotiation.cc:407] Connection header received
0928 02:46:32.288400 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.288403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.288449 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.288557 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.289667 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.290488 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.291892 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.292231 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.292337 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.295407 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.295410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.295412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.295438 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.297166 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.297805 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.297808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.297810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.297898 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.298249 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.298252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.298253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.298527 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.298616 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.299218 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.299357 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:46:32.300379 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.286963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:45044)
0928 02:46:32.287510 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.287555 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.288204 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.288661 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.288667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.289159 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.289529 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.289549 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.290624 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.290627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.291762 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.291773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.292379 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.292381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.292537 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.293391 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.293411 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.295180 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.297273 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.297280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.297290 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.297695 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.297989 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.297992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.297993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.298155 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.298626 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.298632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.298981 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.299160 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.299532 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":125,"spinlock_wait_cycles":6400,"thread_start_us":149,"threads_started":1}
I20260928 02:46:32.300379 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.286962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.28.248.188:45851)
0928 02:46:32.287441 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.287456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.287490 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:32.288187 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.288203 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.288437 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.288653 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.289536 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.290788 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.291892 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.292464 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.292543 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.295932 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.295937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.295948 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.295967 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.297701 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.298169 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.298176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.298181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.298260 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.298528 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.298530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.298531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.298972 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.299165 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.299385 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.299899 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":341,"spinlock_wait_cycles":39296,"thread_start_us":205,"threads_started":1}
I20260928 02:46:32.302120 4046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.188@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
I20260928 02:46:32.310108 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.302651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:50420)
0928 02:46:32.302787 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.302802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.302876 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.303251 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.303254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.303382 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.303478 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.303482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.304357 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.304359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.304982 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.304990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.305397 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.305399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.305469 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.305884 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.305894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.307159 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.308847 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.308851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.308852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.309071 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.309320 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.309322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.309324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.309400 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.309735 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.309737 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.309812 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.309891 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.309995 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 02:46:32.310161 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.302778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.28.248.189:43751)
0928 02:46:32.303057 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.303060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.303072 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:32.303109 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.303111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.303150 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.303227 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.303574 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.304267 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.305079 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.305307 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.305385 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.307286 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.307288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.307291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.307314 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.308752 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.309166 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.309168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.309169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.309242 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.309480 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.309482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.309483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.309650 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.309725 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.309943 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.310042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":127,"threads_started":1}
I20260928 02:46:32.311129 4046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.188@KRBTEST.COM for oryx/127.28.248.188@KRBTEST.COM
I20260928 02:46:32.317898 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.311592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.0.0.1:51262)
0928 02:46:32.311712 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.311727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.311791 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.311978 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.311980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.312099 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.312163 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.312167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.312886 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.312888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.313364 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.313371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.313761 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.313763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.313818 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.314199 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.314209 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.315326 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.316757 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.316759 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.316760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.316941 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.317173 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.317175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.317176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.317245 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.317546 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.317548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.317613 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.317689 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.317777 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:46:32.317963 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.311657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.28.248.188:45851)
0928 02:46:32.311769 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.311773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.311784 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:32.311855 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.311857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.311888 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.311948 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.312250 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.312789 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.313452 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.313675 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.313746 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.315441 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.315444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.315446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.315465 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.316665 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.317029 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.317031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.317032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.317098 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.317318 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.317320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.317321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.317472 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.317544 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.317742 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.317827 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:46:32.321065 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Bootstrap starting.
I20260928 02:46:32.323345 4046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:32.324127 4046 log.cc:824] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:32.326175 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: No bootstrap required, opened a new log
I20260928 02:46:32.329205 4046 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } }
I20260928 02:46:32.329437 4046 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:32.329501 4046 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f8fee6b4de43fc900574449d5cf91d, State: Initialized, Role: FOLLOWER
I20260928 02:46:32.329941 4046 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } }
I20260928 02:46:32.330415 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } } }
I20260928 02:46:32.330549 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:32.331192 4046 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:46:32.336072 4063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:32.336179 4063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:32.339636 3886 master_runner.cc:428] Detected that this master a9f8fee6b4de43fc900574449d5cf91d is joining an existing cluster
I20260928 02:46:32.339704 3886 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.248.188:45851
I20260928 02:46:32.351172 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.342279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:45058)
0928 02:46:32.342590 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.342605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.342714 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.343153 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.343155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.343397 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.343568 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.343575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.344702 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.344707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.345648 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.345657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.346248 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.346251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.346333 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.346971 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.346985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.347531 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.349562 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.349565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.349566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.349770 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.350149 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.350151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.350152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.350232 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.350676 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.350678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.350772 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.350854 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.351013 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":21,"thread_start_us":107,"threads_started":1}
I20260928 02:46:32.351267 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.342146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.28.248.190:43167)
0928 02:46:32.342278 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.342283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.342697 (+ 414us) server_negotiation.cc:407] Connection header received
0928 02:46:32.342885 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.342888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.342943 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.343000 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.343710 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.344573 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.345782 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.346121 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.346232 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.347670 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.347673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.347676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.347705 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.349442 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.349954 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.349960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.349962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.350056 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.350321 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.350324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.350325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.350578 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.350673 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.350934 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.351108 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:46:32.353770 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3382
I20260928 02:46:32.361091 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:42913
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:43167
--webserver_port=36451
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.190@KRBTEST.COM for oryx/127.28.248.188@KRBTEST.COM
I20260928 02:46:32.365799 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.342819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.28.248.188:45851)
0928 02:46:32.342915 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.342921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.343196 (+ 275us) server_negotiation.cc:407] Connection header received
0928 02:46:32.343366 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.343369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.343401 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.343448 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.344046 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.344518 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.346072 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.346379 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.346479 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.362405 (+ 15926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.362409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.362412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.362444 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.364089 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.364533 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.364536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.364538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.364637 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.364886 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.364889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.364890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.365130 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.365212 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.365509 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.365620 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":41}
I20260928 02:46:32.366233 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.342767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.0.0.1:51276)
0928 02:46:32.343129 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.343144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.343273 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.343597 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.343601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.343848 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.343937 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.343942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.344646 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.344651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.345967 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.345975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.346504 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.346508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.346609 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.361610 (+ 15001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.361625 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.362265 (+ 640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.364205 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.364208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.364210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.364440 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.364730 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.364733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.364735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.364805 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.365255 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.365258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.365341 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.365944 (+ 603us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.366073 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":29,"spinlock_wait_cycles":4096,"thread_start_us":74,"threads_started":1}
I20260928 02:46:32.372159 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.355686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.0.0.1:51280)
0928 02:46:32.355804 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.355818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.355895 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.357456 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.357462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.357696 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.357843 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.357853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.358939 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.358946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.359942 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.359956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.360568 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.360574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.360709 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.361584 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.361605 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.363344 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.369648 (+ 6304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.369652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.369655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.370266 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.370607 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.370610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.370612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.370700 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.371357 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.371361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.371481 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.371567 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.371976 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:46:32.372756 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.356862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.28.248.188:45851)
0928 02:46:32.357157 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.357166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.357179 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:32.357247 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.357250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.357301 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.357432 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.357969 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.358816 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.360070 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.360443 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.360544 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.366664 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.366674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.366680 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.366714 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.369495 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.370385 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.370388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.370390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.370486 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.370805 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.370808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.370810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.371004 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.372350 (+ 1346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.372435 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.372552 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260928 02:46:32.374019 3320 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.248.188:45851: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d"
member_type: NON_VOTER
last_known_addr {
host: "127.28.248.188"
port: 45851
}
attrs {
promote: true
}
}
dest_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970"
cas_config_opid_index: 5
I20260928 02:46:32.374691 3320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } }
I20260928 02:46:32.375625 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.342214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.28.248.189:43751)
0928 02:46:32.342341 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.342348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.343187 (+ 839us) server_negotiation.cc:407] Connection header received
0928 02:46:32.343379 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.343382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.343433 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.343484 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.363758 (+ 20274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.364652 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.367737 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.368224 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.368333 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.370801 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.370807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.370811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.370847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.372742 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.373967 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.373985 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.373989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.374098 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.374609 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.374612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.374614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.374829 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.374897 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.375334 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.375472 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:46:32.377069 4020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:32.378592 4020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Refusing update from remote peer ab9e3d501cae4f29af1d1f6aaeb76970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:46:32.379189 3883 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.379304 3881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:32.381536 3883 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.248.188:45851
I20260928 02:46:32.381719 3883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.381829 3883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:32.382234 3886 master.cc:537] Master@127.28.248.188:45851 shutting down...
I20260928 02:46:32.387377 4020 raft_consensus.cc:858] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } }
W20260928 02:46:32.387508 4020 raft_consensus.cc:1565] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } }
I20260928 02:46:32.389925 4020 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c32337296f7d48c0bc020e48af6629b5 (127.28.248.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.391728 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.376025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:50434)
0928 02:46:32.376123 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.376138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.376212 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.382615 (+ 6403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.382618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.382639 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:32.382738 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.382747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.388686 (+ 5939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.388691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.390010 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.390030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.391255 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.391259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.391362 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.391421 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.391499 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.391592 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:46:32.392307 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.376070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.28.248.189:43751)
0928 02:46:32.382216 (+ 6146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.382223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.382242 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:46:32.382302 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.382308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.382374 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.382474 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:32.382865 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.383712 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.390208 (+ 6496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.390903 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.391041 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.391183 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.392061 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.392183 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6028}
I20260928 02:46:32.392270 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9e3d501cae4f29af1d1f6aaeb76970. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.392335 4074 mvcc.cc:205] Tried to move back new op lower bound from 7334148468646395904 to 7334148468516761600. Current Snapshot: MvccSnapshot[applied={T|T < 7334148468555960320}]
I20260928 02:46:32.392400 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:46:32.392761 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.392825 3938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Refusing update from remote peer ab9e3d501cae4f29af1d1f6aaeb76970: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:46:32.392848 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:46:32.392922 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } }
I20260928 02:46:32.392990 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:46:32.393309 3881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:46:32.394414 3938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.395413 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.342381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:50426)
0928 02:46:32.343100 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.343114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.343273 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.343612 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.343615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.361091 (+ 17476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.363598 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.363608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.364805 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.364811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.367165 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.367180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.368524 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.368529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.369643 (+ 1114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.370114 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.370134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.370613 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.373541 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.373547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.373551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.373832 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.374386 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.374391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.374393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.374492 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.375069 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.375072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.375164 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.391770 (+ 16606us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.392132 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"spinlock_wait_cycles":7296,"thread_start_us":86,"threads_started":1}
I20260928 02:46:32.395790 3886 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 02:46:32.396049 3886 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 02:46:32.396003 3967 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9e3d501cae4f29af1d1f6aaeb76970. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.396116 3967 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.396145 3886 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: stopping tablet replica
I20260928 02:46:32.396198 3967 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.396270 3967 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.397572 3881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: NON_VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: true } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.397707 3881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:32.398353 3304 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:46:32.399396 3304 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 -> Peer a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188:45851): Couldn't send request to peer a9f8fee6b4de43fc900574449d5cf91d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:46:32.399955 3883 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: attempting to promote NON_VOTER c32337296f7d48c0bc020e48af6629b5 to VOTER
I20260928 02:46:32.400379 3883 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } }
I20260928 02:46:32.401888 3938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 LEARNER]: Refusing update from remote peer ab9e3d501cae4f29af1d1f6aaeb76970: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:46:32.402608 3881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:46:32.404834 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.404511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.0.0.1:51284)
0928 02:46:32.404684 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.404755 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:45851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:46:32.404884 3881 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c32337296f7d48c0bc020e48af6629b5 (127.28.248.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.406102 3881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.406229 3881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:32.406282 3883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.406373 3883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:32.406555 3304 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 -> Peer a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188:45851): Couldn't send request to peer a9f8fee6b4de43fc900574449d5cf91d. Status: Network error: Client connection negotiation failed: client connection to 127.28.248.188:45851: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:46:32.406790 3938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c32337296f7d48c0bc020e48af6629b5 (127.28.248.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.408241 4082 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.408366 4082 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:32.415020 3886 master.cc:552] Master@127.28.248.188:45851 shutdown complete.
I20260928 02:46:32.415099 3886 master_runner.cc:305] Clearing existing system tablet
I20260928 02:46:32.415323 3886 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.415396 3886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.416550 3886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:32.417436 4085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.417677 3886 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.417829 3886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:32.417868 3886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
uuid: "a9f8fee6b4de43fc900574449d5cf91d"
format_stamp: "Formatted at 2026-09-28 02:46:32 on dist-test-slave-r238"
server_key: "c98693c80a060fccb96351549de6fa5e"
server_key_iv: "a0bf3294d3fd627a59a5b0e31a791d9f"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.419297 3886 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:32.421249 3886 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:46:32.421356 3886 log.cc:1196] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:46:32.421614 3886 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Deleting consensus metadata
I20260928 02:46:32.421789 3886 master_runner.cc:315] Copying system tablet from 127.28.248.190:43167
I20260928 02:46:32.422945 3886 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.248.190:43167
I20260928 02:46:32.431816 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.423504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:45062)
0928 02:46:32.423845 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.423856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.423998 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.424664 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.424670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.424949 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.425048 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.425055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.425822 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.425826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.426430 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.426438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.427331 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.427333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.427399 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.427947 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.427959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.428287 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.430161 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.430165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.430167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.430372 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.430681 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.430683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.430685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.430773 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.431250 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.431256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.431351 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.431461 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.431622 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":80,"threads_started":1}
I20260928 02:46:32.431838 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.423576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.28.248.190:43167)
0928 02:46:32.423708 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.423713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.423940 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:46:32.424236 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.424239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.424294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.424384 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.425172 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.425707 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.426558 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.426830 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.426947 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.428429 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.428433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.428436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.428461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.430030 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.430480 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.430484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.430486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.430574 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.430870 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.430873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.430875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.431054 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.431184 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.431514 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.431623 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:32.432664 3340 tablet_copy_service.cc:140] P ab9e3d501cae4f29af1d1f6aaeb76970: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a9f8fee6b4de43fc900574449d5cf91d ({username='oryx', principal='oryx/127.28.248.188@KRBTEST.COM'} at 127.0.0.1:45062)
I20260928 02:46:32.432770 3340 tablet_copy_service.cc:161] P ab9e3d501cae4f29af1d1f6aaeb76970: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a9f8fee6b4de43fc900574449d5cf91d at {username='oryx', principal='oryx/127.28.248.188@KRBTEST.COM'} at 127.0.0.1:45062: session id = a9f8fee6b4de43fc900574449d5cf91d-00000000000000000000000000000000
I20260928 02:46:32.433781 3340 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:46:32.434959 3886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:46:32.436977 3886 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: tablet copy: Starting download of 0 data blocks...
I20260928 02:46:32.437183 3886 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: tablet copy: Starting download of 1 WAL segments...
I20260928 02:46:32.438541 3886 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:46:32.440239 3340 tablet_copy_service.cc:342] P ab9e3d501cae4f29af1d1f6aaeb76970: Request end of tablet copy session a9f8fee6b4de43fc900574449d5cf91d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.248.188@KRBTEST.COM'} at 127.0.0.1:45062
I20260928 02:46:32.440337 3340 tablet_copy_service.cc:433] P ab9e3d501cae4f29af1d1f6aaeb76970: ending tablet copy session a9f8fee6b4de43fc900574449d5cf91d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a9f8fee6b4de43fc900574449d5cf91d
I20260928 02:46:32.451099 3886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:32.453049 4093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.453045 4092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.453413 4095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.456791 3886 server_base.cc:1034] running on GCE node
I20260928 02:46:32.457008 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:32.457333 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:32.458478 3886 hybrid_clock.cc:648] HybridClock initialized: now 1790563592458481 us; error 51 us; skew 500 ppm
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:32.460023 3886 init.cc:377] Logged in from keytab as oryx/127.28.248.188@KRBTEST.COM (short username oryx)
I20260928 02:46:32.460534 3886 webserver.cc:460] Webserver started at http://127.28.248.188:42247/ using document root <none> and password file <none>
I20260928 02:46:32.460837 3886 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.460897 3886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.462136 3886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:32.463120 4102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.463347 3886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:32.463443 3886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
uuid: "a9f8fee6b4de43fc900574449d5cf91d"
format_stamp: "Formatted at 2026-09-28 02:46:32 on dist-test-slave-r238"
server_key: "c98693c80a060fccb96351549de6fa5e"
server_key_iv: "a0bf3294d3fd627a59a5b0e31a791d9f"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.463555 3886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.474514 3645 heartbeater.cc:503] Master 127.28.248.190:43167 was elected leader, sending a full tablet report...
I20260928 02:46:32.478418 3886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:32.480481 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:32.485692 3886 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.188:45851
I20260928 02:46:32.485705 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.188:45851 every 8 connection(s)
I20260928 02:46:32.486245 3886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
I20260928 02:46:32.488241 4155 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:32.489167 4155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Bootstrap starting.
I20260928 02:46:32.495714 4155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:32.496066 4155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Bootstrap complete.
I20260928 02:46:32.496556 4155 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } }
I20260928 02:46:32.496728 4155 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9f8fee6b4de43fc900574449d5cf91d, State: Initialized, Role: LEARNER
I20260928 02:46:32.496847 4155 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } }
I20260928 02:46:32.497175 4158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.497336 4158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: LEARNER
I20260928 02:46:32.497334 4155 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:32.501201 4169 catalog_manager.cc:1259] Loaded cluster ID: 078e495dc1a7446c98c0039f003aeff4
I20260928 02:46:32.501277 4169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: loading cluster ID for follower catalog manager: success
I20260928 02:46:32.502804 4169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: acquiring CA information for follower catalog manager: success
I20260928 02:46:32.503950 4169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:46:32.525390 4073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:32.525827 4073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:32.525892 4073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:32.532366 4073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:32.532470 4073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:32.532562 4073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:32.540832 4073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:42913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=36451
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:32.542356 4073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:32.543687 4073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:32.554445 4175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.554492 4177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.554517 4073 server_base.cc:1034] running on GCE node
W20260928 02:46:32.554462 4174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.555086 4073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:32.555685 4073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:32.556838 4073 hybrid_clock.cc:648] HybridClock initialized: now 1790563592556830 us; error 42 us; skew 500 ppm
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:32.559677 4073 init.cc:377] Logged in from keytab as oryx/127.28.248.129@KRBTEST.COM (short username oryx)
I20260928 02:46:32.560706 4073 webserver.cc:460] Webserver started at http://127.28.248.129:36451/ using document root <none> and password file <none>
I20260928 02:46:32.561286 4073 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.561403 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.565652 4073 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:32.567898 4184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.569443 4073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:32.569581 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "e05df32e19e040589e50754445d84c74"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "ded701b5bfcfb65c0bc5ef1ae055654a"
server_key_iv: "0549f5f824e1236cdc3f5e00efffdbed"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.570168 4073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.590829 4073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:32.594283 4073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:32.594493 4073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:32.595268 4073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:32.596289 4073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:32.596357 4073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.596423 4073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:32.596470 4073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.606482 4073 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:42913
I20260928 02:46:32.606482 4297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:42913 every 8 connection(s)
I20260928 02:46:32.607666 4073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:32.609592 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 4073
I20260928 02:46:32.609786 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3516
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.188@KRBTEST.COM
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
I20260928 02:46:32.625787 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:40821
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:43167
--webserver_port=33597
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:32.628443 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.610469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:60049 (local address 127.28.248.190:43167)
0928 02:46:32.610662 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.610669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.612833 (+ 2164us) server_negotiation.cc:407] Connection header received
0928 02:46:32.613525 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.613531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.613603 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.613662 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.615815 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.616771 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619056 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.619388 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619720 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.624255 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.624261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.624265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.624302 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.626412 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.627067 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.627070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.627072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.627163 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.627489 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.627492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.627494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.627694 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.627764 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.628144 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.628272 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:46:32.628448 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.611612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:50035 (local address 127.28.248.188:45851)
0928 02:46:32.611899 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.611904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.612531 (+ 627us) server_negotiation.cc:407] Connection header received
0928 02:46:32.613502 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.613506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.613567 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.613628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.615801 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.616876 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619218 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.619521 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619654 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.622860 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.622866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.622872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.622911 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.625264 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.626522 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.626526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.626528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.626634 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.627082 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.627085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.627087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.627350 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.627428 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.628144 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.628275 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260928 02:46:32.628508 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.611882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:54245 (local address 127.28.248.189:43751)
0928 02:46:32.612045 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.612052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.612233 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:46:32.613505 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.613508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.613567 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.613627 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.615801 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.616685 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619201 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.619557 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619650 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.624313 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.624317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.624320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.624354 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.626275 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.626803 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.626809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.626813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.626911 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.627240 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.627246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.627250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.627500 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.627570 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.628152 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.628291 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:46:32.630379 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.610337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.28.248.129:54245)
0928 02:46:32.612079 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.612122 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.613237 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.614058 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.614069 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.614755 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.615622 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.615639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.616858 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.616861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.619072 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.619082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.619789 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.619899 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.620837 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.620860 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.624172 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.626407 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.626414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.626419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.626685 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.627030 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.627035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.627040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.627130 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.627652 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.627656 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.627857 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.628754 (+ 897us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.629326 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":630,"spinlock_wait_cycles":17792,"thread_start_us":164,"threads_started":1}
I20260928 02:46:32.630375 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.611474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.28.248.129:50035)
0928 02:46:32.612442 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.612454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.613237 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.614058 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.614069 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.614760 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.615622 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.615639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.617015 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.617018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.619071 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.619082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.619697 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.619895 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.620903 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.620915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.622617 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.625430 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.625438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.625456 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.626373 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.626738 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.626744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.626749 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.626971 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.627518 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.627527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.627857 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.629613 (+ 1756us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.629757 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":564,"spinlock_wait_cycles":14848,"thread_start_us":85,"threads_started":1}
I20260928 02:46:32.630376 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.612121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:60049)
0928 02:46:32.612746 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.612759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.613269 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.614059 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.614071 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.614742 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.615618 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.615641 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.616925 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.616930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.618675 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.618693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.619850 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.619853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.619993 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.620859 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.620884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.623734 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.626526 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.626529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.626531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.626943 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.627281 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.627283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.627285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.627380 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.627838 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.627841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.627911 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.628730 (+ 819us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.629227 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":461,"spinlock_wait_cycles":16640,"thread_start_us":145,"threads_started":1}
I20260928 02:46:32.631760 4299 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43167
I20260928 02:46:32.631861 4298 heartbeater.cc:347] Connected to a master server at 127.28.248.188:45851
I20260928 02:46:32.632014 4299 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:32.632011 4298 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:32.632040 4300 heartbeater.cc:347] Connected to a master server at 127.28.248.189:43751
I20260928 02:46:32.632161 4300 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:32.632696 4299 heartbeater.cc:511] Master 127.28.248.190:43167 requested a full tablet report, sending...
I20260928 02:46:32.632697 4300 heartbeater.cc:511] Master 127.28.248.189:43751 requested a full tablet report, sending...
I20260928 02:46:32.633293 4298 heartbeater.cc:511] Master 127.28.248.188:45851 requested a full tablet report, sending...
I20260928 02:46:32.634009 3320 ts_manager.cc:194] Re-registered known tserver with Master: e05df32e19e040589e50754445d84c74 (127.28.248.129:42913)
I20260928 02:46:32.634308 3920 ts_manager.cc:194] Registered new tserver with Master: e05df32e19e040589e50754445d84c74 (127.28.248.129:42913)
I20260928 02:46:32.634487 4119 ts_manager.cc:194] Registered new tserver with Master: e05df32e19e040589e50754445d84c74 (127.28.248.129:42913)
I20260928 02:46:32.634732 3320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:60049
I20260928 02:46:32.702675 3779 heartbeater.cc:503] Master 127.28.248.190:43167 was elected leader, sending a full tablet report...
W20260928 02:46:32.745647 4306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:32.745926 4306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:32.745972 4306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:32.750352 4306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:32.750434 4306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:32.750559 4306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:32.755950 4306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:40821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=33597
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:32.757120 4306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:32.758193 4306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:46:32.768081 4306 server_base.cc:1034] running on GCE node
W20260928 02:46:32.768031 4311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.768040 4312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.768092 4314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.768903 4306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:32.769609 4306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:32.770790 4306 hybrid_clock.cc:648] HybridClock initialized: now 1790563592770775 us; error 36 us; skew 500 ppm
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:32.773694 4306 init.cc:377] Logged in from keytab as oryx/127.28.248.130@KRBTEST.COM (short username oryx)
I20260928 02:46:32.774662 4306 webserver.cc:460] Webserver started at http://127.28.248.130:33597/ using document root <none> and password file <none>
I20260928 02:46:32.775208 4306 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.775277 4306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.779170 4306 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:32.781230 4321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.782580 4306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:32.782698 4306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "4f4e91168a294f54942159198b96ef85"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "9ff41d246d89c1dcf4d4fb267b982fe5"
server_key_iv: "939ed619b33c51ca29c503eb9ee08dca"
server_key_version: "encryptionkey@0"
I20260928 02:46:32.783272 4306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:32.801962 4306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:32.805016 4306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:32.805173 4306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:32.805699 4306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:32.806512 4306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:32.806565 4306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.806605 4306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:32.806622 4306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:32.815922 4306 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:40821
I20260928 02:46:32.815948 4434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:40821 every 8 connection(s)
I20260928 02:46:32.817055 4306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:32.823745 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 4306
I20260928 02:46:32.823909 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3650
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.188@KRBTEST.COM
I20260928 02:46:32.831861 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:38611
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:43167
--webserver_port=43291
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:32.834671 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.821036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:44551 (local address 127.28.248.189:43751)
0928 02:46:32.821175 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.821179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.821191 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:32.821673 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.821676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.821733 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.821811 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.823221 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.824098 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.826331 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.826784 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.826882 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.829887 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.829891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.829895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.829927 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.832001 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.832689 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.832696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.832701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.832797 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.833213 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.833217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.833222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.833506 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.833590 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.834323 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.834470 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:46:32.834704 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.818998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:42717 (local address 127.28.248.188:45851)
0928 02:46:32.819136 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.819141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.820590 (+ 1449us) server_negotiation.cc:407] Connection header received
0928 02:46:32.821704 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.821708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.821769 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.821821 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.823261 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.824057 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.826333 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.826774 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.826844 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.830211 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.830217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.830223 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.830260 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.831998 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.833070 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.833075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.833079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.833178 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.833507 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.833511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.833512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.833736 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.833833 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.834335 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.834498 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:32.836278 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.820648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.28.248.130:44551)
0928 02:46:32.820934 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.820949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.821475 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.822000 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.822008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.822441 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.823078 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.823096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.824256 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.824261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.826180 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.826195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.827077 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.827083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.827216 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.827932 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.827957 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.829674 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.832134 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.832143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.832164 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.832546 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.832910 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.832915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.832920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.833105 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.833659 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.833666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.834018 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.834958 (+ 940us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.835329 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":348,"thread_start_us":107,"threads_started":1}
I20260928 02:46:32.836575 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.819498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.28.248.130:42717)
0928 02:46:32.820357 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.820419 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.821450 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.822001 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.822008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.822479 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.823093 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.823100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.824248 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.824252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.826185 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.826199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.827025 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.827027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.827169 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.828005 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.828025 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.830050 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.832693 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.832696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.832698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.832937 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.833303 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.833307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.833310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.833407 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.833850 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.833856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.834030 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.835295 (+ 1265us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.835449 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":463,"spinlock_wait_cycles":43392,"thread_start_us":143,"threads_started":1}
I20260928 02:46:32.838114 4435 heartbeater.cc:347] Connected to a master server at 127.28.248.188:45851
I20260928 02:46:32.838171 4437 heartbeater.cc:347] Connected to a master server at 127.28.248.189:43751
I20260928 02:46:32.838382 4435 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:32.838382 4437 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:32.838745 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.818998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:45111 (local address 127.28.248.190:43167)
0928 02:46:32.819135 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.819140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.820534 (+ 1394us) server_negotiation.cc:407] Connection header received
0928 02:46:32.821704 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.821708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.821766 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.821821 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:32.823315 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.824098 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.832215 (+ 8117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.832536 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.832659 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.835052 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:32.835056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:32.835058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:32.835085 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:32.836832 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.837388 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.837391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.837393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.837501 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:32.837814 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:32.837818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:32.837820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:32.838049 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:32.838136 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.838478 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.838607 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:32.839061 4435 heartbeater.cc:511] Master 127.28.248.188:45851 requested a full tablet report, sending...
I20260928 02:46:32.839246 4437 heartbeater.cc:511] Master 127.28.248.189:43751 requested a full tablet report, sending...
I20260928 02:46:32.839272 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.819498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:45111)
0928 02:46:32.820357 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.820419 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.821449 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.822001 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.822008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.822488 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:32.823090 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.823100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.824357 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.824360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.832050 (+ 7690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.832063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.832829 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.832834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.832951 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.833448 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:32.833466 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:32.834868 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:32.836978 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.836983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.836988 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.837232 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.837620 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:32.837624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:32.837626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:32.837714 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:32.838192 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:32.838196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:32.838286 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.838872 (+ 586us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.839050 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":6617,"spinlock_wait_cycles":74112,"thread_start_us":151,"threads_started":1}
I20260928 02:46:32.840230 4436 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43167
I20260928 02:46:32.840301 4436 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:32.840384 3920 ts_manager.cc:194] Registered new tserver with Master: 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821)
I20260928 02:46:32.840402 4119 ts_manager.cc:194] Registered new tserver with Master: 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821)
I20260928 02:46:32.840456 4436 heartbeater.cc:511] Master 127.28.248.190:43167 requested a full tablet report, sending...
I20260928 02:46:32.841161 3320 ts_manager.cc:194] Re-registered known tserver with Master: 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821)
I20260928 02:46:32.842027 3320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:45111
I20260928 02:46:32.889421 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.885175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.28.248.188:45851)
0928 02:46:32.885302 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:32.885307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:32.885322 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:32.885487 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:32.885489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:32.885551 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:32.885628 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:32.885971 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.886577 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.888172 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.888803 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.888954 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.889169 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:32.889206 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 02:46:32.889274 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:46:32.889453 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.885061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.0.0.1:51290)
0928 02:46:32.885228 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.885247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:32.885365 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:32.885660 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:32.885663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:32.885682 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:32.885815 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.885823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.886722 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.886727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.887998 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:32.888016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:32.888970 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:32.888974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:32.889084 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:32.889114 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:46:32.889189 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:46:32.889296 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:46:32.890291 4129 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:46:32.890666 4158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9e3d501cae4f29af1d1f6aaeb76970. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:32.890864 4158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: LEARNER
W20260928 02:46:32.960877 4442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:32.961247 4442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:32.961300 4442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:32.965726 4442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:32.965790 4442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:32.965855 4442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:32.971700 4442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:38611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=43291
--tserver_master_addrs=127.28.248.190:43167,127.28.248.189:43751,127.28.248.188:45851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:32.972929 4442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:32.974148 4442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:46:32.985267 4442 server_base.cc:1034] running on GCE node
W20260928 02:46:32.985173 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.985215 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:32.985173 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:32.985909 4442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:32.986725 4442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:32.987895 4442 hybrid_clock.cc:648] HybridClock initialized: now 1790563592987847 us; error 69 us; skew 500 ppm
Sep 28 02:46:32 dist-test-slave-r238 krb5kdc[3275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:32.991008 4442 init.cc:377] Logged in from keytab as oryx/127.28.248.131@KRBTEST.COM (short username oryx)
I20260928 02:46:32.992106 4442 webserver.cc:460] Webserver started at http://127.28.248.131:43291/ using document root <none> and password file <none>
I20260928 02:46:32.992765 4442 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:32.992846 4442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:32.996966 4442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:32.999275 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:33.000559 4442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:33.000725 4442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc"
format_stamp: "Formatted at 2026-09-28 02:46:31 on dist-test-slave-r238"
server_key: "d3861405cbecf5e8f9ed105731162689"
server_key_iv: "c4ae635ad29357e3409223244bbf3817"
server_key_version: "encryptionkey@0"
I20260928 02:46:33.001262 4442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:33.018270 4442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:33.022456 4442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:33.022673 4442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:33.023481 4442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:33.024753 4442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:33.024819 4442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:33.024895 4442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:33.024935 4442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:33.035305 4442 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:38611
I20260928 02:46:33.035322 4570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:38611 every 8 connection(s)
I20260928 02:46:33.036355 4442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:33.041831 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 4442
Sep 28 02:46:33 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
Sep 28 02:46:33 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.189@KRBTEST.COM
Sep 28 02:46:33 dist-test-slave-r238 krb5kdc[3275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563592, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.188@KRBTEST.COM
I20260928 02:46:33.054373 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.038282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:45999 (local address 127.28.248.189:43751)
0928 02:46:33.038405 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.038410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.039439 (+ 1029us) server_negotiation.cc:407] Connection header received
0928 02:46:33.040700 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.040702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.040748 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.040847 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.042416 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.043188 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045035 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.045374 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045468 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.050131 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.050136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.050163 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.050194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.051899 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.052514 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.052517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.052519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.052600 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.053019 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.053022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.053024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.053219 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.053299 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.054025 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.054157 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:46:33.055022 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.040022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:37533 (local address 127.28.248.188:45851)
0928 02:46:33.040149 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.040154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.040168 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:33.040968 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.040971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.041024 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.041109 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.042401 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.043231 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045978 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.046335 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.046413 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.051138 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.051144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.051148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.051179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.053057 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.053621 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.053625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.053627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.053708 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.054035 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.054039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.054042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.054274 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.054347 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.054721 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.054845 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:46:33.056186 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.038632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.28.248.131:45999)
0928 02:46:33.039331 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.039345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.040316 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.040907 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.040915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.041431 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:33.042231 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.042250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.043379 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.043382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.044926 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.044937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.045637 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.046220 (+ 578us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.047004 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:33.047022 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:33.049956 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:33.052018 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:33.052025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:33.052042 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:33.052400 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:33.052737 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:33.052740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:33.052742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:33.052916 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:33.053383 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:33.053392 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:33.053725 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.054552 (+ 827us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.054962 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":509,"spinlock_wait_cycles":61056,"thread_start_us":154,"threads_started":1}
I20260928 02:46:33.056201 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.039678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.28.248.131:37533)
0928 02:46:33.039932 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.039946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.040316 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.041149 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.041152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.041469 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:33.042228 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.042248 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.043382 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.043386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.045846 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.045855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.046593 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.046597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.046690 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.047096 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:33.047112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:33.051012 (+ 3900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:33.053196 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:33.053200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:33.053202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:33.053496 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:33.053830 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:33.053835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:33.053839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:33.053925 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:33.054436 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:33.054439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:33.054545 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.055165 (+ 620us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.055307 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":1574,"spinlock_wait_cycles":4352,"thread_start_us":86,"threads_started":1}
I20260928 02:46:33.057495 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.038208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:34457 (local address 127.28.248.190:43167)
0928 02:46:33.038348 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.038353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.039304 (+ 951us) server_negotiation.cc:407] Connection header received
0928 02:46:33.040541 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.040544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.040604 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.040724 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.042402 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.043231 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045071 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.045406 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045479 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.049406 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.049410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.049412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.049439 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.052330 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.055646 (+ 3316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.055649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.055651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.055733 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.056032 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.056035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.056036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.056249 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.056338 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.057238 (+ 900us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.057359 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:33.057636 4571 heartbeater.cc:347] Connected to a master server at 127.28.248.188:45851
I20260928 02:46:33.057873 4573 heartbeater.cc:347] Connected to a master server at 127.28.248.189:43751
I20260928 02:46:33.057999 4573 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:33.058529 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.044191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.28.248.190:43167)
0928 02:46:33.045150 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.045163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.045174 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:33.045209 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.045212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.045254 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.045336 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.046393 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.047301 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.048672 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.048936 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.049052 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.050842 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.050846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.050848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.050875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.054356 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.056896 (+ 2540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.056902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.056904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.056996 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.057344 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.057347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.057349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.057571 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.057684 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.058235 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.058354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":79,"threads_started":1}
I20260928 02:46:33.058614 4573 heartbeater.cc:511] Master 127.28.248.189:43751 requested a full tablet report, sending...
I20260928 02:46:33.059791 3920 ts_manager.cc:194] Registered new tserver with Master: c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611)
I20260928 02:46:33.057905 4571 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:33.060012 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.044063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.28.248.189:43751)
0928 02:46:33.045021 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.045025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.049407 (+ 4382us) server_negotiation.cc:407] Connection header received
0928 02:46:33.049793 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.049797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.049853 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.049921 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.051032 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.051883 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.053018 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.053347 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.053413 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.055176 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.055180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.055182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.055204 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.056941 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.058496 (+ 1555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.058500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.058501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.058572 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.058913 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.058918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.058919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.059156 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.059308 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.059641 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.059765 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":72,"threads_started":1}
I20260928 02:46:33.060035 4571 heartbeater.cc:511] Master 127.28.248.188:45851 requested a full tablet report, sending...
I20260928 02:46:33.060844 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.038632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:34457)
0928 02:46:33.039149 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.039199 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.040316 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.040906 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.040913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.041459 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:33.042228 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.042248 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.043382 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.043386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.044943 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.044948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.045636 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.045639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.045918 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.046625 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:33.046644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:33.049204 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:33.055011 (+ 5807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:33.055014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:33.055015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:33.055532 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:33.055838 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:33.055841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:33.055843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:33.055920 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:33.056814 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:33.056818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:33.056905 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.060511 (+ 3606us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.060678 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":537,"spinlock_wait_cycles":7808,"thread_start_us":146,"threads_started":1}
I20260928 02:46:33.060962 4119 ts_manager.cc:194] Registered new tserver with Master: c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611)
I20260928 02:46:33.061432 4572 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43167
I20260928 02:46:33.061517 4572 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:33.061677 4572 heartbeater.cc:511] Master 127.28.248.190:43167 requested a full tablet report, sending...
I20260928 02:46:33.062256 3320 ts_manager.cc:194] Re-registered known tserver with Master: c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611)
I20260928 02:46:33.062791 3320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:34457
I20260928 02:46:33.062597 3319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:33.066470 3319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:33.069727 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: NON_VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: true } } }
I20260928 02:46:33.069903 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:33.070420 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.044671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.28.248.188:45851)
0928 02:46:33.049685 (+ 5014us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.049689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.054555 (+ 4866us) server_negotiation.cc:407] Connection header received
0928 02:46:33.058126 (+ 3571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.058132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.058185 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.058261 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.063067 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.063719 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.064686 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.064970 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.065079 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.066522 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.066526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.066529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.066560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.068366 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.068908 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.068911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.068913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.069069 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.069356 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.069359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.069361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.069608 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.069741 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.070090 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.070226 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4933,"thread_start_us":88,"threads_started":1}
I20260928 02:46:33.072294 4589 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: attempting to promote NON_VOTER a9f8fee6b4de43fc900574449d5cf91d to VOTER
I20260928 02:46:33.072660 4589 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } }
I20260928 02:46:33.074146 4128 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 LEARNER]: Refusing update from remote peer ab9e3d501cae4f29af1d1f6aaeb76970: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 02:46:33.074582 4590 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:46:33.075590 3938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Refusing update from remote peer ab9e3d501cae4f29af1d1f6aaeb76970: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 02:46:33.075978 3882 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 02:46:33.076599 3882 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.078104 4590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.078222 4590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:33.078222 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.078439 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:33.078361 3938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.079242 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.079352 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:33.080075 4599 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.080266 4599 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:33.081799 4129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.083850 4158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:33.083967 4158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:33.086733 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.080749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42913 (local address 127.0.0.1:36520)
0928 02:46:33.081050 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.081066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.081161 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.082522 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.082525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.082540 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:33.082606 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.082611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.083909 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.083912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.085019 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.085032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.086291 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.086295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.086384 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.086411 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.086509 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.086578 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":3,"spinlock_wait_cycles":3584,"thread_start_us":132,"threads_started":1}
I20260928 02:46:33.086810 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.081527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.28.248.129:42913)
0928 02:46:33.082133 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.082148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.082175 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:46:33.082236 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.082240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.082391 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.082557 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:33.082685 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.083770 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.085323 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.086175 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.086278 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.086448 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.086557 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.086662 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":71,"threads_started":1}
I20260928 02:46:33.087553 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.080444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:38611 (local address 127.0.0.1:44250)
0928 02:46:33.081026 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.081042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.081150 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.082237 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.082240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.082259 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:33.082370 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.082377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.083909 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.083912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.086005 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.086017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.087138 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.087141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.087277 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.087300 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.087365 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.087432 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":939,"spinlock_wait_cycles":7040}
I20260928 02:46:33.087618 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.081025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.28.248.131:38611)
0928 02:46:33.081514 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.081527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.081553 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:46:33.081634 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.081639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.081890 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.082067 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:33.082502 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.083773 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.086157 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.087030 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.087149 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.087323 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.087389 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.087479 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":120,"threads_started":1}
I20260928 02:46:33.090464 4505 tablet_service.cc:1467] Processing CreateTablet for tablet 9e27cd6d810c49448d55435727b0d3da (DEFAULT_TABLE table=test-table [id=059cc02f7e1542489f82d187e558d162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:33.090466 4232 tablet_service.cc:1467] Processing CreateTablet for tablet 9e27cd6d810c49448d55435727b0d3da (DEFAULT_TABLE table=test-table [id=059cc02f7e1542489f82d187e558d162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:33.091516 4232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e27cd6d810c49448d55435727b0d3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:33.091542 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e27cd6d810c49448d55435727b0d3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:33.092245 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.081378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40821 (local address 127.0.0.1:55990)
0928 02:46:33.084606 (+ 3228us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.084655 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.084747 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.088101 (+ 3354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.088103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.088120 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:33.088191 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.088196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.089631 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.089634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.090784 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.090797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.091822 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.091825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.091919 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.091947 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.092035 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.092107 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3152,"thread_start_us":79,"threads_started":1}
I20260928 02:46:33.092290 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.081547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.28.248.130:40821)
0928 02:46:33.086688 (+ 5141us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.086700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.086723 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:46:33.086800 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.086807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.087059 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.087227 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:33.088388 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.089475 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.090948 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.091694 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.091821 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.091950 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.092064 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.092150 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5030,"thread_start_us":128,"threads_started":1}
I20260928 02:46:33.095887 4369 tablet_service.cc:1467] Processing CreateTablet for tablet 9e27cd6d810c49448d55435727b0d3da (DEFAULT_TABLE table=test-table [id=059cc02f7e1542489f82d187e558d162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:33.097069 4369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e27cd6d810c49448d55435727b0d3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:33.099327 4605 tablet_bootstrap.cc:492] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Bootstrap starting.
I20260928 02:46:33.100142 4606 tablet_bootstrap.cc:492] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Bootstrap starting.
I20260928 02:46:33.101593 4605 tablet_bootstrap.cc:654] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:33.102313 4606 tablet_bootstrap.cc:654] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:33.102353 4605 log.cc:824] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:33.103107 4606 log.cc:824] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:33.105934 4609 tablet_bootstrap.cc:492] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Bootstrap starting.
I20260928 02:46:33.106504 4605 tablet_bootstrap.cc:492] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: No bootstrap required, opened a new log
I20260928 02:46:33.106686 4605 ts_tablet_manager.cc:1397] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 02:46:33.106753 4606 tablet_bootstrap.cc:492] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: No bootstrap required, opened a new log
I20260928 02:46:33.106891 4606 ts_tablet_manager.cc:1397] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:46:33.108158 4609 tablet_bootstrap.cc:654] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:33.109462 4609 log.cc:824] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:33.110831 4606 raft_consensus.cc:348] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.111078 4606 raft_consensus.cc:374] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:33.111132 4606 raft_consensus.cc:729] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80fcd1bb9b9414db6aa73ec3034a7dc, State: Initialized, Role: FOLLOWER
I20260928 02:46:33.111244 4605 raft_consensus.cc:348] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.111491 4605 raft_consensus.cc:374] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:33.111546 4605 raft_consensus.cc:729] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05df32e19e040589e50754445d84c74, State: Initialized, Role: FOLLOWER
I20260928 02:46:33.111593 4609 tablet_bootstrap.cc:492] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: No bootstrap required, opened a new log
I20260928 02:46:33.111778 4609 ts_tablet_manager.cc:1397] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:46:33.111680 4606 consensus_queue.cc:260] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.111974 4605 consensus_queue.cc:260] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.112759 4606 ts_tablet_manager.cc:1428] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260928 02:46:33.112934 4605 ts_tablet_manager.cc:1428] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260928 02:46:33.113023 4572 heartbeater.cc:503] Master 127.28.248.190:43167 was elected leader, sending a full tablet report...
I20260928 02:46:33.113417 4299 heartbeater.cc:503] Master 127.28.248.190:43167 was elected leader, sending a full tablet report...
I20260928 02:46:33.117403 4609 raft_consensus.cc:348] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.117661 4609 raft_consensus.cc:374] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:33.117717 4609 raft_consensus.cc:729] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4e91168a294f54942159198b96ef85, State: Initialized, Role: FOLLOWER
I20260928 02:46:33.118134 4609 consensus_queue.cc:260] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.119176 4609 ts_tablet_manager.cc:1428] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 02:46:33.120360 4436 heartbeater.cc:503] Master 127.28.248.190:43167 was elected leader, sending a full tablet report...
I20260928 02:46:33.177685 4612 raft_consensus.cc:482] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:33.177942 4612 raft_consensus.cc:504] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.179152 4612 leader_election.cc:288] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611), 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821)
I20260928 02:46:33.183203 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.179536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:36725 (local address 127.28.248.130:40821)
0928 02:46:33.179670 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.179676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.179853 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:46:33.180009 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.180012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.180068 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.180156 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:33.180458 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.181017 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.182064 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.182505 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.182629 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.182681 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.182980 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.183056 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:33.183223 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.179539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:45837 (local address 127.28.248.131:38611)
0928 02:46:33.179746 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.179753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.179794 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:46:33.179984 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.179987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.180028 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.180107 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:33.180388 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.181146 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.182123 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.182528 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.182655 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.182706 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.183015 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.183090 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260928 02:46:33.183248 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.179416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:38611 (local address 127.28.248.129:45837)
0928 02:46:33.179715 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.179730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.179848 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.180129 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.180133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.180148 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:33.180276 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.180282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.181267 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.181270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.182000 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.182008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.182650 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.182654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.182795 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.182864 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.182997 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.183077 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":35,"thread_start_us":89,"threads_started":1}
I20260928 02:46:33.183260 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.179418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40821 (local address 127.28.248.129:36725)
0928 02:46:33.179761 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.179772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:33.179894 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:33.180166 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:33.180170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:33.180198 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:33.180322 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.180327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.181127 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.181132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.181928 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:33.181937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.182625 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.182629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:33.182745 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.182871 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 02:46:33.182963 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:46:33.183066 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":129,"spinlock_wait_cycles":768,"thread_start_us":75,"threads_started":1}
I20260928 02:46:33.184108 4389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e27cd6d810c49448d55435727b0d3da" candidate_uuid: "e05df32e19e040589e50754445d84c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4e91168a294f54942159198b96ef85" is_pre_election: true
I20260928 02:46:33.184173 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e27cd6d810c49448d55435727b0d3da" candidate_uuid: "e05df32e19e040589e50754445d84c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" is_pre_election: true
I20260928 02:46:33.184444 4389 raft_consensus.cc:2454] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e05df32e19e040589e50754445d84c74 in term 0.
I20260928 02:46:33.184463 4525 raft_consensus.cc:2454] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e05df32e19e040589e50754445d84c74 in term 0.
I20260928 02:46:33.184931 4188 leader_election.cc:302] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4e91168a294f54942159198b96ef85, e05df32e19e040589e50754445d84c74; no voters:
I20260928 02:46:33.185225 4612 raft_consensus.cc:2781] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:33.185308 4612 raft_consensus.cc:482] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:33.185351 4612 raft_consensus.cc:3037] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:33.186417 4612 raft_consensus.cc:504] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.186806 4612 leader_election.cc:288] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [CANDIDATE]: Term 1 election: Requested vote from peers c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611), 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821)
I20260928 02:46:33.187150 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e27cd6d810c49448d55435727b0d3da" candidate_uuid: "e05df32e19e040589e50754445d84c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc"
I20260928 02:46:33.187278 4389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e27cd6d810c49448d55435727b0d3da" candidate_uuid: "e05df32e19e040589e50754445d84c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4e91168a294f54942159198b96ef85"
I20260928 02:46:33.187296 4525 raft_consensus.cc:3037] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:33.187420 4389 raft_consensus.cc:3037] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:33.188441 4525 raft_consensus.cc:2454] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e05df32e19e040589e50754445d84c74 in term 1.
I20260928 02:46:33.188467 4389 raft_consensus.cc:2454] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e05df32e19e040589e50754445d84c74 in term 1.
I20260928 02:46:33.188748 4185 leader_election.cc:302] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80fcd1bb9b9414db6aa73ec3034a7dc, e05df32e19e040589e50754445d84c74; no voters:
I20260928 02:46:33.188938 4612 raft_consensus.cc:2781] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:33.189164 4612 raft_consensus.cc:686] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 1 LEADER]: Becoming Leader. State: Replica: e05df32e19e040589e50754445d84c74, State: Running, Role: LEADER
I20260928 02:46:33.189553 4612 consensus_queue.cc:237] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } }
I20260928 02:46:33.193202 3320 catalog_manager.cc:5495] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 reported cstate change: term changed from 0 to 1, leader changed from <none> to e05df32e19e040589e50754445d84c74 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "e05df32e19e040589e50754445d84c74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:33.216348 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.213175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.28.248.129:42913)
0928 02:46:33.213311 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.213322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.213337 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:33.213412 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.213414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.213469 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.213558 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:33.213766 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.214289 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.215076 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.215277 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.215350 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.215605 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:33.216041 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:33.216132 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.216189 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.216238 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:46:33.222312 4525 raft_consensus.cc:1261] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Refusing update from remote peer e05df32e19e040589e50754445d84c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:33.222312 4389 raft_consensus.cc:1261] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Refusing update from remote peer e05df32e19e040589e50754445d84c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:33.223083 4616 consensus_queue.cc:1035] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:33.223279 4612 consensus_queue.cc:1035] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:33.235261 4621 mvcc.cc:205] Tried to move back new op lower bound from 7334148477831266304 to 7334148477707558912. Current Snapshot: MvccSnapshot[applied={T|T < 7334148477831266304}]
I20260928 02:46:33.238087 4620 mvcc.cc:205] Tried to move back new op lower bound from 7334148477831266304 to 7334148477707558912. Current Snapshot: MvccSnapshot[applied={T|T < 7334148477831266304}]
I20260928 02:46:33.242447 3320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:33.242632 3320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:33.247156 3320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab9e3d501cae4f29af1d1f6aaeb76970: Sending DeleteTablet for 3 replicas of tablet 9e27cd6d810c49448d55435727b0d3da
I20260928 02:46:33.248371 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e27cd6d810c49448d55435727b0d3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:33 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:44250
I20260928 02:46:33.249001 4369 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e27cd6d810c49448d55435727b0d3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:33 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260928 02:46:33.249107 4628 tablet_replica.cc:330] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: stopping tablet replica
I20260928 02:46:33.249493 4628 raft_consensus.cc:2229] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:33.249799 4232 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e27cd6d810c49448d55435727b0d3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:33 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:36520
I20260928 02:46:33.249855 4628 raft_consensus.cc:2258] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:33.251350 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3288
I20260928 02:46:33.252010 4628 ts_tablet_manager.cc:1905] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:33.255546 4628 ts_tablet_manager.cc:1918] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:33.255666 4628 log.cc:1196] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/9e27cd6d810c49448d55435727b0d3da
I20260928 02:46:33.255999 4628 ts_tablet_manager.cc:1939] T 9e27cd6d810c49448d55435727b0d3da P c80fcd1bb9b9414db6aa73ec3034a7dc: Deleting consensus metadata
I20260928 02:46:33.256891 4629 tablet_replica.cc:330] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: stopping tablet replica
I20260928 02:46:33.257251 4629 raft_consensus.cc:2229] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:33.257596 4629 raft_consensus.cc:2258] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:33.259881 4629 ts_tablet_manager.cc:1905] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:46:33.260382 4459 connection.cc:535] server connection from 127.0.0.1:44250 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:46:33.261600 4631 tablet_replica.cc:330] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: stopping tablet replica
I20260928 02:46:33.263899 4631 raft_consensus.cc:2229] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:33.264305 4631 raft_consensus.cc:2258] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:33.264590 4629 ts_tablet_manager.cc:1918] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:33.266028 4631 ts_tablet_manager.cc:1905] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:33.267198 4629 log.cc:1196] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/9e27cd6d810c49448d55435727b0d3da
I20260928 02:46:33.267616 4629 ts_tablet_manager.cc:1939] T 9e27cd6d810c49448d55435727b0d3da P 4f4e91168a294f54942159198b96ef85: Deleting consensus metadata
I20260928 02:46:33.269833 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.249892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.28.248.189:43751)
0928 02:46:33.250117 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.250124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.258132 (+ 8008us) server_negotiation.cc:407] Connection header received
0928 02:46:33.258255 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.258261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.258353 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.258453 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.259684 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.260586 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.261917 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.262264 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.262366 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.265503 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.265508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.265511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.265541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.267516 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.268167 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.268171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.268173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.268257 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.268552 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.268557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.268559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.268806 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.268923 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.269409 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.269573 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
W20260928 02:46:33.270146 4323 connection.cc:441] server connection from 127.0.0.1:55990 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55990 (request call id 1) could send its response
I20260928 02:46:33.271276 4631 ts_tablet_manager.cc:1918] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:33.271379 4631 log.cc:1196] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/9e27cd6d810c49448d55435727b0d3da
I20260928 02:46:33.271705 4631 ts_tablet_manager.cc:1939] T 9e27cd6d810c49448d55435727b0d3da P e05df32e19e040589e50754445d84c74: Deleting consensus metadata
W20260928 02:46:33.272809 4185 connection.cc:441] server connection from 127.0.0.1:36520 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36520 (request call id 1) could send its response
I20260928 02:46:33.277060 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.265587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.28.248.189:43751)
0928 02:46:33.265710 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.265715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.266457 (+ 742us) server_negotiation.cc:407] Connection header received
0928 02:46:33.266649 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.266654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.266706 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.266769 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.267680 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.268490 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.270451 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.270825 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.270912 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.272936 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.272944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.272948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.272983 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.274914 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.275473 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.275477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.275479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.275568 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.275886 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.275890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.275892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.276111 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.276223 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.276686 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.276831 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:46:33.282721 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.266122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.28.248.188:45851)
0928 02:46:33.266257 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.266262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.271701 (+ 5439us) server_negotiation.cc:407] Connection header received
0928 02:46:33.273211 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.273215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.273256 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.273303 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.274144 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.274915 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.276141 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.276485 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.276553 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.278448 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.278452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.278455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.278480 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.280461 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.281138 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.281141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.281143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.281224 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.281513 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.281516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.281518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.281779 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.281915 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.282340 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.282557 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:33.283524 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.249999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.28.248.188:45851)
0928 02:46:33.250107 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:33.250113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:33.272216 (+ 22103us) server_negotiation.cc:407] Connection header received
0928 02:46:33.272439 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:33.272446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:33.272523 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:33.272588 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:33.273696 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.274617 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.275887 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:33.276237 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:33.276323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:33.279030 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:33.279034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:33.279037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:33.279061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:33.281019 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.281805 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.281813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.281817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.281931 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:33.282435 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:33.282441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:33.282443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:33.282721 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:33.282847 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:33.283199 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:46:33.283327 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:46:34.236575 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.236138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:51397)
0928 02:46:34.236389 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.236478 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
W20260928 02:46:34.237038 4572 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:43167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.241400 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.240915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:60211)
0928 02:46:34.241213 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.241310 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":118,"threads_started":1}
W20260928 02:46:34.241732 4299 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:43167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.242308 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.241867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:39075)
0928 02:46:34.242082 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.242166 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
W20260928 02:46:34.242669 4436 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:43167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.748123 4641 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab9e3d501cae4f29af1d1f6aaeb76970)
I20260928 02:46:34.748387 4641 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } }
I20260928 02:46:34.749435 4641 leader_election.cc:288] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167), c32337296f7d48c0bc020e48af6629b5 (127.28.248.189:43751)
I20260928 02:46:34.750175 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.749636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48686)
0928 02:46:34.749954 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.750041 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
W20260928 02:46:34.750414 4106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.750727 4645 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab9e3d501cae4f29af1d1f6aaeb76970)
I20260928 02:46:34.750846 4645 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } }
I20260928 02:46:34.751277 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.751052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48696)
0928 02:46:34.751164 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.751217 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:46:34.751390 4645 leader_election.cc:288] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167), a9f8fee6b4de43fc900574449d5cf91d (127.28.248.188:45851)
W20260928 02:46:34.751741 4106 leader_election.cc:334] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167): Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.751977 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.751461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48706)
0928 02:46:34.751817 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.751883 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":69,"threads_started":1}
W20260928 02:46:34.752164 3915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.753018 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.752785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48708)
0928 02:46:34.752888 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.752938 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 02:46:34.753196 3915 leader_election.cc:334] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167): Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.754038 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.749744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.28.248.189:43751)
0928 02:46:34.750046 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.750050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.750151 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:46:34.750275 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.750279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.750349 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.750459 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.750774 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.751553 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.752985 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.753533 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.753625 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.753688 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.753891 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.753960 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":60,"threads_started":1}
I20260928 02:46:34.754079 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.749706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:43751 (local address 127.0.0.1:58692)
0928 02:46:34.750057 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.750070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.750170 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.750471 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.750475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.750491 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.750618 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.750624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.751684 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.751688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.752841 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.752851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.753666 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.753670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.753759 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.753784 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.753863 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.753951 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":45,"threads_started":1}
I20260928 02:46:34.754545 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f8fee6b4de43fc900574449d5cf91d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c32337296f7d48c0bc020e48af6629b5" is_pre_election: true
I20260928 02:46:34.754769 3938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f8fee6b4de43fc900574449d5cf91d in term 1.
I20260928 02:46:34.755155 4103 leader_election.cc:302] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9f8fee6b4de43fc900574449d5cf91d, c32337296f7d48c0bc020e48af6629b5; no voters: ab9e3d501cae4f29af1d1f6aaeb76970
I20260928 02:46:34.755405 4641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:46:34.755470 4641 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab9e3d501cae4f29af1d1f6aaeb76970)
I20260928 02:46:34.755510 4641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:46:34.756152 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.751554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:45851 (local address 127.0.0.1:45582)
0928 02:46:34.752044 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.752513 (+ 469us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.752635 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.752986 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.752989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.753005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.753104 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.753112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.753962 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.753965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.755018 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.755032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.755859 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.755862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.755928 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.755947 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.756016 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.756065 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":80,"threads_started":1}
I20260928 02:46:34.756212 4649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.751685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.28.248.188:45851)
0928 02:46:34.751921 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.751930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.752594 (+ 664us) server_negotiation.cc:407] Connection header received
0928 02:46:34.752768 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.752771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.752812 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.752888 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.753200 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.753847 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.755130 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.755736 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.755864 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.755907 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.756043 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.756110 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260928 02:46:34.756656 4129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c32337296f7d48c0bc020e48af6629b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a9f8fee6b4de43fc900574449d5cf91d" is_pre_election: true
I20260928 02:46:34.756953 4641 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } }
I20260928 02:46:34.757190 4129 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c32337296f7d48c0bc020e48af6629b5 in current term 2: Already voted for candidate a9f8fee6b4de43fc900574449d5cf91d in this term.
I20260928 02:46:34.757354 4641 leader_election.cc:288] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [CANDIDATE]: Term 2 election: Requested vote from peers ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167), c32337296f7d48c0bc020e48af6629b5 (127.28.248.189:43751)
I20260928 02:46:34.757647 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.757451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48712)
0928 02:46:34.757537 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.757589 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:46:34.757624 3912 leader_election.cc:302] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c32337296f7d48c0bc020e48af6629b5; no voters: a9f8fee6b4de43fc900574449d5cf91d, ab9e3d501cae4f29af1d1f6aaeb76970
I20260928 02:46:34.757664 3938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9f8fee6b4de43fc900574449d5cf91d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c32337296f7d48c0bc020e48af6629b5"
I20260928 02:46:34.757771 3938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:46:34.758255 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.758050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48716)
0928 02:46:34.758157 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.758204 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:46:34.758370 4106 leader_election.cc:334] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167): Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
I20260928 02:46:34.758941 3938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f8fee6b4de43fc900574449d5cf91d in term 2.
I20260928 02:46:34.759060 4645 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:46:34.759281 4103 leader_election.cc:302] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9f8fee6b4de43fc900574449d5cf91d, c32337296f7d48c0bc020e48af6629b5; no voters: ab9e3d501cae4f29af1d1f6aaeb76970
I20260928 02:46:34.759435 4641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:46:34.759636 4641 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [term 2 LEADER]: Becoming Leader. State: Replica: a9f8fee6b4de43fc900574449d5cf91d, State: Running, Role: LEADER
I20260928 02:46:34.759902 4641 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } }
I20260928 02:46:34.760990 4651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f8fee6b4de43fc900574449d5cf91d. Latest consensus state: current_term: 2 leader_uuid: "a9f8fee6b4de43fc900574449d5cf91d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:34.761081 4651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:34.761358 4653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:34.763613 4653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:34.764015 4653 catalog_manager.cc:1259] Loaded cluster ID: 078e495dc1a7446c98c0039f003aeff4
I20260928 02:46:34.764060 4653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:34.764664 4653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:34.765062 4653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Loaded TSK: 0
I20260928 02:46:34.765750 4653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:34.804306 4118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:34.807829 4118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:34.811018 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.810734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48720)
0928 02:46:34.810853 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.810925 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:46:34.811182 3938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [term 2 FOLLOWER]: Refusing update from remote peer a9f8fee6b4de43fc900574449d5cf91d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:46:34.811762 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.811525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.0.0.1:48722)
0928 02:46:34.811637 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.811692 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:46:34.811791 4641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 02:46:34.812076 4106 consensus_peers.cc:487] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d -> Peer ab9e3d501cae4f29af1d1f6aaeb76970 (127.28.248.190:43167): Couldn't send request to peer ab9e3d501cae4f29af1d1f6aaeb76970. Status: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:46:34.813530 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f8fee6b4de43fc900574449d5cf91d. Latest consensus state: current_term: 2 leader_uuid: "a9f8fee6b4de43fc900574449d5cf91d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:34.813692 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:34.814741 4651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a9f8fee6b4de43fc900574449d5cf91d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:34.814877 4651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:34.815265 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a9f8fee6b4de43fc900574449d5cf91d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ab9e3d501cae4f29af1d1f6aaeb76970" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43167 } } peers { permanent_uuid: "c32337296f7d48c0bc020e48af6629b5" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 43751 } attrs { promote: false } } peers { permanent_uuid: "a9f8fee6b4de43fc900574449d5cf91d" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 45851 } attrs { promote: false } } }
I20260928 02:46:34.815392 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32337296f7d48c0bc020e48af6629b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:34.826391 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.822013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.28.248.131:38611)
0928 02:46:34.822277 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.822281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.822295 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:34.822404 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.822407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.822446 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.822549 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.822860 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.823588 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.825283 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.825868 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.826005 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.826061 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.826195 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.826263 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260928 02:46:34.826391 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.821648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:38611 (local address 127.0.0.1:60010)
0928 02:46:34.821732 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.821745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.821824 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.822558 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.822561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.822578 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.822723 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.822729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.823691 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.823697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.824840 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.824850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.825988 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.825991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.826082 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.826106 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.826171 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.826248 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":81}
I20260928 02:46:34.826576 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.821165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40821 (local address 127.0.0.1:37564)
0928 02:46:34.821312 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.821328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.821411 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.822568 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.822571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.822587 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.822707 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.822713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.823846 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.823849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.825342 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.825350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.826199 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.826203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.826291 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.826317 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.826364 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.826436 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":215}
I20260928 02:46:34.826912 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.821648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42913 (local address 127.0.0.1:43702)
0928 02:46:34.822756 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.822771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.822917 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.823203 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.823207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.823224 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.823297 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.823303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.824302 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.824306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.825569 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.825578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.826519 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.826522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.826609 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.826630 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.826716 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.826790 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":332,"thread_start_us":86,"threads_started":1}
I20260928 02:46:34.826987 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.821749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.28.248.129:42913)
0928 02:46:34.822024 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.822031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.822840 (+ 809us) server_negotiation.cc:407] Connection header received
0928 02:46:34.823030 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.823033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.823088 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.823175 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.823409 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.824190 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.825690 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.826312 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.826420 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.826465 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.826762 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.826842 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260928 02:46:34.827148 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.821296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37564 (local address 127.28.248.130:40821)
0928 02:46:34.822342 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.822346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.822357 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:34.822401 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.822403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.822442 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.822557 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.822820 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.823685 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.825462 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.826064 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.826851 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.826907 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.826946 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.827025 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":67,"threads_started":1}
I20260928 02:46:34.827628 4505 tablet_service.cc:1467] Processing CreateTablet for tablet cb1591813e1a443ab4ce986e1b1649e4 (DEFAULT_TABLE table=test-table [id=ade77d4373ee4faebcc1a99ddac757e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:34.827627 4369 tablet_service.cc:1467] Processing CreateTablet for tablet cb1591813e1a443ab4ce986e1b1649e4 (DEFAULT_TABLE table=test-table [id=ade77d4373ee4faebcc1a99ddac757e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:34.827948 4369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1591813e1a443ab4ce986e1b1649e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:34.827953 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1591813e1a443ab4ce986e1b1649e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:34.830564 4664 tablet_bootstrap.cc:492] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Bootstrap starting.
I20260928 02:46:34.830758 4232 tablet_service.cc:1467] Processing CreateTablet for tablet cb1591813e1a443ab4ce986e1b1649e4 (DEFAULT_TABLE table=test-table [id=ade77d4373ee4faebcc1a99ddac757e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:34.831022 4232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb1591813e1a443ab4ce986e1b1649e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:34.831393 4664 tablet_bootstrap.cc:654] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:34.832047 4665 tablet_bootstrap.cc:492] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Bootstrap starting.
I20260928 02:46:34.832681 4664 tablet_bootstrap.cc:492] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: No bootstrap required, opened a new log
I20260928 02:46:34.832759 4664 ts_tablet_manager.cc:1397] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:34.833191 4664 raft_consensus.cc:348] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.833294 4664 raft_consensus.cc:374] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:34.833316 4664 raft_consensus.cc:729] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4e91168a294f54942159198b96ef85, State: Initialized, Role: FOLLOWER
I20260928 02:46:34.833318 4667 tablet_bootstrap.cc:492] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: Bootstrap starting.
I20260928 02:46:34.833444 4664 consensus_queue.cc:260] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.833765 4664 ts_tablet_manager.cc:1428] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:34.834231 4667 tablet_bootstrap.cc:654] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:34.835204 4665 tablet_bootstrap.cc:654] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:34.835700 4667 tablet_bootstrap.cc:492] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: No bootstrap required, opened a new log
I20260928 02:46:34.835798 4667 ts_tablet_manager.cc:1397] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:34.835978 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.834214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:57437)
0928 02:46:34.835814 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.835878 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1527,"thread_start_us":92,"threads_started":1}
I20260928 02:46:34.836225 4667 raft_consensus.cc:348] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.836537 4435 heartbeater.cc:503] Master 127.28.248.188:45851 was elected leader, sending a full tablet report...
I20260928 02:46:34.836798 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.836404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:40037)
0928 02:46:34.836681 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.836728 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260928 02:46:34.836344 4667 raft_consensus.cc:374] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:34.836844 4667 raft_consensus.cc:729] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05df32e19e040589e50754445d84c74, State: Initialized, Role: FOLLOWER
I20260928 02:46:34.837019 4667 consensus_queue.cc:260] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.837488 4667 ts_tablet_manager.cc:1428] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:34.838505 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.838122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:38771)
0928 02:46:34.838352 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.838405 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
I20260928 02:46:34.839133 4298 heartbeater.cc:503] Master 127.28.248.188:45851 was elected leader, sending a full tablet report...
I20260928 02:46:34.839257 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.838915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:34423)
0928 02:46:34.839141 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.839191 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260928 02:46:34.839383 4665 tablet_bootstrap.cc:492] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: No bootstrap required, opened a new log
I20260928 02:46:34.839479 4665 ts_tablet_manager.cc:1397] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 02:46:34.839869 4665 raft_consensus.cc:348] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.840000 4665 raft_consensus.cc:374] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:34.840046 4665 raft_consensus.cc:729] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80fcd1bb9b9414db6aa73ec3034a7dc, State: Initialized, Role: FOLLOWER
I20260928 02:46:34.840183 4665 consensus_queue.cc:260] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.840534 4665 ts_tablet_manager.cc:1428] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:34.841481 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.840991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:58949)
0928 02:46:34.841225 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.841278 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260928 02:46:34.841965 4571 heartbeater.cc:503] Master 127.28.248.188:45851 was elected leader, sending a full tablet report...
I20260928 02:46:34.842151 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.841890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:56011)
0928 02:46:34.842053 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.842099 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:46:34.843773 4668 raft_consensus.cc:482] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:34.843895 4668 raft_consensus.cc:504] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.845009 4668 leader_election.cc:288] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611), e05df32e19e040589e50754445d84c74 (127.28.248.129:42913)
I20260928 02:46:34.850368 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.845195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:38611 (local address 127.28.248.130:54031)
0928 02:46:34.845380 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.845393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.845491 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.845830 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.845832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.845842 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.846006 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.846014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.847045 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.847049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.848830 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.848842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.849780 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.849784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.849964 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.850044 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.850124 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.850230 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":362,"spinlock_wait_cycles":1792}
I20260928 02:46:34.850451 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.845433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:38347 (local address 127.28.248.129:42913)
0928 02:46:34.845578 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.845582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.845596 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:34.845643 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.845646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.845693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.845771 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.846125 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.846903 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.848983 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.849602 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.849723 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.849774 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.850235 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.850328 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:46:34.850461 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.845546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:54031 (local address 127.28.248.131:38611)
0928 02:46:34.845645 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.845649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.845660 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:34.845696 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.845699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.845746 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.845817 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:34.846125 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.846903 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.848991 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.849600 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.849723 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.849774 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.850275 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.850351 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:46:34.850701 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.845188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42913 (local address 127.28.248.130:38347)
0928 02:46:34.845336 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.845351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.845458 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.845804 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.845810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.845830 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:34.846013 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.846019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.847045 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.847049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.848830 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.848842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.849780 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.849784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.849996 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.850045 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.850124 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.850621 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":719}
I20260928 02:46:34.851229 4252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1591813e1a443ab4ce986e1b1649e4" candidate_uuid: "4f4e91168a294f54942159198b96ef85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05df32e19e040589e50754445d84c74" is_pre_election: true
I20260928 02:46:34.851269 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1591813e1a443ab4ce986e1b1649e4" candidate_uuid: "4f4e91168a294f54942159198b96ef85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" is_pre_election: true
I20260928 02:46:34.851409 4525 raft_consensus.cc:2454] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f4e91168a294f54942159198b96ef85 in term 0.
I20260928 02:46:34.851481 4252 raft_consensus.cc:2454] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f4e91168a294f54942159198b96ef85 in term 0.
I20260928 02:46:34.851750 4322 leader_election.cc:302] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4e91168a294f54942159198b96ef85, c80fcd1bb9b9414db6aa73ec3034a7dc; no voters:
I20260928 02:46:34.852003 4668 raft_consensus.cc:2781] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:34.852062 4668 raft_consensus.cc:482] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:34.852100 4668 raft_consensus.cc:3037] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:34.853123 4668 raft_consensus.cc:504] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.853459 4668 leader_election.cc:288] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [CANDIDATE]: Term 1 election: Requested vote from peers c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611), e05df32e19e040589e50754445d84c74 (127.28.248.129:42913)
I20260928 02:46:34.853754 4525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1591813e1a443ab4ce986e1b1649e4" candidate_uuid: "4f4e91168a294f54942159198b96ef85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc"
I20260928 02:46:34.853801 4252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb1591813e1a443ab4ce986e1b1649e4" candidate_uuid: "4f4e91168a294f54942159198b96ef85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05df32e19e040589e50754445d84c74"
I20260928 02:46:34.853878 4525 raft_consensus.cc:3037] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:34.853896 4252 raft_consensus.cc:3037] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:34.854995 4525 raft_consensus.cc:2454] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f4e91168a294f54942159198b96ef85 in term 1.
I20260928 02:46:34.855023 4252 raft_consensus.cc:2454] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f4e91168a294f54942159198b96ef85 in term 1.
I20260928 02:46:34.855265 4324 leader_election.cc:302] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4e91168a294f54942159198b96ef85, e05df32e19e040589e50754445d84c74; no voters:
I20260928 02:46:34.855441 4668 raft_consensus.cc:2781] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:34.855666 4668 raft_consensus.cc:686] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 1 LEADER]: Becoming Leader. State: Replica: 4f4e91168a294f54942159198b96ef85, State: Running, Role: LEADER
I20260928 02:46:34.855930 4668 consensus_queue.cc:237] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } }
I20260928 02:46:34.856359 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.856152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:35535)
0928 02:46:34.856251 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.856301 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:46:34.858513 4119 catalog_manager.cc:5495] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f4e91168a294f54942159198b96ef85 (127.28.248.130). New cstate: current_term: 1 leader_uuid: "4f4e91168a294f54942159198b96ef85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f4e91168a294f54942159198b96ef85" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 } health_report { overall_health: UNKNOWN } } }
W20260928 02:46:34.860808 4301 tablet.cc:2369] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:34.877301 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.873653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.28.248.130:40821)
0928 02:46:34.873807 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.873814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.873849 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:46:34.873937 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.873939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.873992 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.874110 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:34.874356 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.874941 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.875789 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.876022 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.876134 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.876397 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:34.876915 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:34.877043 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.877100 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.877156 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:34.880879 4252 raft_consensus.cc:1261] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Refusing update from remote peer 4f4e91168a294f54942159198b96ef85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:34.881547 4677 consensus_queue.cc:1035] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e05df32e19e040589e50754445d84c74" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:34.881666 4525 raft_consensus.cc:1261] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Refusing update from remote peer 4f4e91168a294f54942159198b96ef85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:34.882082 4677 consensus_queue.cc:1035] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80fcd1bb9b9414db6aa73ec3034a7dc" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:34.883411 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.883096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:53667)
0928 02:46:34.883269 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.883330 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:46:34.884547 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.884277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:45831)
0928 02:46:34.884417 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.884472 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:46:34.887734 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.887242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.130:52367)
0928 02:46:34.887599 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.887659 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
I20260928 02:46:34.887887 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.887640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.129:45689)
0928 02:46:34.887774 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.887821 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:46:34.887887 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.887640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43167 (local address 127.28.248.131:45083)
0928 02:46:34.887774 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.887821 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:46:34.888123 4681 mvcc.cc:205] Tried to move back new op lower bound from 7334148484625108992 to 7334148484532932608. Current Snapshot: MvccSnapshot[applied={T|T < 7334148484625108992}]
I20260928 02:46:34.895639 4119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:34.895813 4119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:34.899626 4119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9f8fee6b4de43fc900574449d5cf91d: Sending DeleteTablet for 3 replicas of tablet cb1591813e1a443ab4ce986e1b1649e4
I20260928 02:46:34.900290 4369 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1591813e1a443ab4ce986e1b1649e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:34 UTC) from {username='oryx', principal='oryx/127.28.248.188@KRBTEST.COM'} at 127.0.0.1:37564
I20260928 02:46:34.900717 4690 tablet_replica.cc:330] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: stopping tablet replica
I20260928 02:46:34.900862 4690 raft_consensus.cc:2229] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:34.900883 4232 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1591813e1a443ab4ce986e1b1649e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:34 UTC) from {username='oryx', principal='oryx/127.28.248.188@KRBTEST.COM'} at 127.0.0.1:43702
I20260928 02:46:34.901115 4505 tablet_service.cc:1514] Processing DeleteTablet for tablet cb1591813e1a443ab4ce986e1b1649e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:34 UTC) from {username='oryx', principal='oryx/127.28.248.188@KRBTEST.COM'} at 127.0.0.1:60010
I20260928 02:46:34.901147 4690 raft_consensus.cc:2258] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:34.901407 4691 tablet_replica.cc:330] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74: stopping tablet replica
I20260928 02:46:34.901579 4691 raft_consensus.cc:2229] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:34.901883 4691 raft_consensus.cc:2258] T cb1591813e1a443ab4ce986e1b1649e4 P e05df32e19e040589e50754445d84c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:34.901918 4690 ts_tablet_manager.cc:1905] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:34.902112 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 4073
I20260928 02:46:34.905952 4690 ts_tablet_manager.cc:1918] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:34.906047 4690 log.cc:1196] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/cb1591813e1a443ab4ce986e1b1649e4
I20260928 02:46:34.906311 4690 ts_tablet_manager.cc:1939] T cb1591813e1a443ab4ce986e1b1649e4 P 4f4e91168a294f54942159198b96ef85: Deleting consensus metadata
I20260928 02:46:34.906945 4692 tablet_replica.cc:330] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: stopping tablet replica
I20260928 02:46:34.906994 4106 catalog_manager.cc:4841] TS 4f4e91168a294f54942159198b96ef85 (127.28.248.130:40821): tablet cb1591813e1a443ab4ce986e1b1649e4 (table test-table [id=ade77d4373ee4faebcc1a99ddac757e7]) successfully deleted
I20260928 02:46:34.907073 4692 raft_consensus.cc:2229] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:34.907199 4692 raft_consensus.cc:2258] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:34.907824 4692 ts_tablet_manager.cc:1905] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:34.909675 4692 ts_tablet_manager.cc:1918] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:34.909771 4692 log.cc:1196] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/cb1591813e1a443ab4ce986e1b1649e4
I20260928 02:46:34.910108 4692 ts_tablet_manager.cc:1939] T cb1591813e1a443ab4ce986e1b1649e4 P c80fcd1bb9b9414db6aa73ec3034a7dc: Deleting consensus metadata
I20260928 02:46:34.910748 4103 catalog_manager.cc:4841] TS c80fcd1bb9b9414db6aa73ec3034a7dc (127.28.248.131:38611): tablet cb1591813e1a443ab4ce986e1b1649e4 (table test-table [id=ade77d4373ee4faebcc1a99ddac757e7]) successfully deleted
I20260928 02:46:34.912925 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 4306
I20260928 02:46:34.913170 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.912905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42913 (local address 127.0.0.1:43714)
0928 02:46:34.913043 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.913093 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:46:34.913694 4105 catalog_manager.cc:4584] TS e05df32e19e040589e50754445d84c74 (127.28.248.129:42913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb1591813e1a443ab4ce986e1b1649e4: Network error: Client connection negotiation failed: client connection to 127.28.248.129:42913: connect: Connection refused (error 111)
I20260928 02:46:34.918993 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 4442
I20260928 02:46:34.925094 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3784
I20260928 02:46:34.931149 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 3886
2026-09-28T02:46:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5372 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:34 dist-test-slave-r238 krb5kdc[4695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:34 dist-test-slave-r238 krb5kdc[4695](info): set up 2 sockets
Sep 28 02:46:34 dist-test-slave-r238 krb5kdc[4695](info): commencing operation
krb5kdc: starting...
W20260928 02:46:35.977295 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab.
Entry for principal oryx/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab.
I20260928 02:46:36.100137 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:37681
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab
--principal=oryx/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:37681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:36.209218 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:36.209520 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:36.209594 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:36.213735 4708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:36.213819 4708 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:46:36.213848 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:36.213891 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:36.213949 4708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:36.213989 4708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:36.219293 4708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:37681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.190.keytab
--principal=oryx/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:36.220398 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:36.221489 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:36.229920 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.229961 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.229928 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:36.230144 4708 server_base.cc:1034] running on GCE node
I20260928 02:46:36.230769 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:36.231810 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:36.232967 4708 hybrid_clock.cc:648] HybridClock initialized: now 1790563596232941 us; error 38 us; skew 500 ppm
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:36.235682 4708 init.cc:377] Logged in from keytab as oryx/127.28.248.190@KRBTEST.COM (short username oryx)
I20260928 02:46:36.236515 4708 webserver.cc:460] Webserver started at http://127.28.248.190:41405/ using document root <none> and password file <none>
I20260928 02:46:36.237102 4708 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:36.237164 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:36.237323 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:36.239259 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "9e0c366c2f6c40fa8d8087fad87fa655"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "71438165a851c1fffb8e9f5943c77cf7"
server_key_iv: "0deb9cff4aab7339248587b132907f9d"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.239771 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "9e0c366c2f6c40fa8d8087fad87fa655"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "71438165a851c1fffb8e9f5943c77cf7"
server_key_iv: "0deb9cff4aab7339248587b132907f9d"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.243392 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 02:46:36.245824 4723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.247124 4708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:36.247253 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "9e0c366c2f6c40fa8d8087fad87fa655"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "71438165a851c1fffb8e9f5943c77cf7"
server_key_iv: "0deb9cff4aab7339248587b132907f9d"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.247360 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:36.275683 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:36.278604 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:36.278807 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:36.286481 4708 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:37681
I20260928 02:46:36.286511 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:37681 every 8 connection(s)
I20260928 02:46:36.287475 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:36.290166 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:36.295349 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Bootstrap starting.
I20260928 02:46:36.296682 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 4708
I20260928 02:46:36.296811 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:36.297080 29666 external_mini_cluster.cc:1428] Setting key 5b69ab4f827bebd5d1a4b57369ed56dd
I20260928 02:46:36.297964 4786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:36.298693 4786 log.cc:824] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:36.300817 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: No bootstrap required, opened a new log
I20260928 02:46:36.304966 4786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37681 } }
I20260928 02:46:36.305260 4786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:36.305332 4786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0c366c2f6c40fa8d8087fad87fa655, State: Initialized, Role: FOLLOWER
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:36.305838 4786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37681 } }
I20260928 02:46:36.305985 4786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:36.306061 4786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:36.306160 4786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:36.307065 4786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37681 } }
I20260928 02:46:36.307397 4786 leader_election.cc:302] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e0c366c2f6c40fa8d8087fad87fa655; no voters:
I20260928 02:46:36.307704 4786 leader_election.cc:288] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:36.307823 4791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:36.308044 4791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [term 1 LEADER]: Becoming Leader. State: Replica: 9e0c366c2f6c40fa8d8087fad87fa655, State: Running, Role: LEADER
I20260928 02:46:36.308346 4791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37681 } }
I20260928 02:46:36.309036 4786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:36.309999 4793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37681 } } }
I20260928 02:46:36.310111 4793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:36.310400 4792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e0c366c2f6c40fa8d8087fad87fa655. Latest consensus state: current_term: 1 leader_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e0c366c2f6c40fa8d8087fad87fa655" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 37681 } } }
I20260928 02:46:36.310585 4792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:36.312817 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.298177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.28.248.190:37681)
0928 02:46:36.298619 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:36.298634 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:36.298667 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:46:36.299308 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:36.299325 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:36.299646 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:36.299912 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:36.300773 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.301776 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.303031 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.303499 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.303610 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.306546 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:36.306551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:36.306563 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:36.306587 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:36.308721 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.309257 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.309262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.309268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.309438 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.309762 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.309768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.309772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.310322 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:36.311305 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:36.311450 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:46:36.311892 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
W20260928 02:46:36.314405 4807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:36.314499 4807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:36.314606 4800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:36.315474 4800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:36.320791 4800 catalog_manager.cc:1347] Generated new cluster ID: 842067b044c045d387bbcf05245a90b5
I20260928 02:46:36.320854 4800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:36.327206 4800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:36.328392 4800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:36.342752 4800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Generated new TSK 0
I20260928 02:46:36.343608 4800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab.
Entry for principal oryx/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab.
I20260928 02:46:36.371310 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37681
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:36.482055 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:36.482295 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:36.482337 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:36.486416 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:36.486471 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:36.486541 4812 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:36.486596 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:36.491727 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:36.492762 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:36.493742 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:36.502928 4820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.502979 4818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:36.503044 4812 server_base.cc:1034] running on GCE node
W20260928 02:46:36.503069 4817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:36.503544 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:36.504190 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:36.505375 4812 hybrid_clock.cc:648] HybridClock initialized: now 1790563596505341 us; error 44 us; skew 500 ppm
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:36.507941 4812 init.cc:377] Logged in from keytab as oryx/127.28.248.129@KRBTEST.COM (short username oryx)
I20260928 02:46:36.508798 4812 webserver.cc:460] Webserver started at http://127.28.248.129:39055/ using document root <none> and password file <none>
I20260928 02:46:36.509356 4812 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:36.509447 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:36.509721 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:36.511858 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "e2f8468eb794442c92d916a922f8f584"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "858098d21657ffe510995f0da3c69c26"
server_key_iv: "d831b1fdc2b4038e60de3cf749654438"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.512404 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "e2f8468eb794442c92d916a922f8f584"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "858098d21657ffe510995f0da3c69c26"
server_key_iv: "d831b1fdc2b4038e60de3cf749654438"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.515905 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:36.518270 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.519428 4812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:36.519572 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "e2f8468eb794442c92d916a922f8f584"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "858098d21657ffe510995f0da3c69c26"
server_key_iv: "d831b1fdc2b4038e60de3cf749654438"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.519683 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:36.545205 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:36.548228 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:36.548432 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:36.549883 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:36.549957 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.550051 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:36.550104 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:36.560745 4812 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:42331
I20260928 02:46:36.560812 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:42331 every 8 connection(s)
I20260928 02:46:36.561599 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:36.564044 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.550855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:45873 (local address 127.28.248.190:37681)
0928 02:46:36.551006 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:36.551011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:36.551967 (+ 956us) server_negotiation.cc:407] Connection header received
0928 02:46:36.553121 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:36.553125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:36.553190 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:36.553299 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:36.554696 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.555593 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.557525 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.557847 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.557932 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.560646 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:36.560651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:36.560653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:36.560685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:36.562340 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.562823 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.562825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.562826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.562896 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.563169 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.563171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.563172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.563329 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:36.563402 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:36.563818 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 02:46:36.563915 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:46:36.564885 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.551291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.28.248.129:45873)
0928 02:46:36.551807 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:36.551855 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:36.552878 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:36.553475 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:36.553483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:36.553915 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:36.554530 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.554548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.555759 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.555766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.557368 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.557383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.558016 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.558020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.558248 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.558995 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:36.559010 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:36.560392 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:36.562452 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.562457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.562470 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.562739 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.562978 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.562979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.562981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.563097 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.563419 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:36.563423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:36.563629 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 02:46:36.563775 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:46:36.564016 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":160,"threads_started":1}
I20260928 02:46:36.566072 4943 heartbeater.cc:347] Connected to a master server at 127.28.248.190:37681
I20260928 02:46:36.566200 4943 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:36.566713 4943 heartbeater.cc:511] Master 127.28.248.190:37681 requested a full tablet report, sending...
I20260928 02:46:36.567554 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 4812
I20260928 02:46:36.567682 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:36.568009 29666 external_mini_cluster.cc:1428] Setting key afaab2f83c7dd5cf3ab3752789ecb60c
I20260928 02:46:36.568310 4739 ts_manager.cc:194] Registered new tserver with Master: e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331)
I20260928 02:46:36.569898 4739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:45873
I20260928 02:46:36.576411 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.568645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:60109 (local address 127.28.248.190:37681)
0928 02:46:36.569061 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:36.569064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:36.569101 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:46:36.569136 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:36.569140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:36.569183 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:36.569264 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:36.569740 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.570612 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.571638 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.571849 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.571964 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.573273 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:36.573276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:36.573278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:36.573298 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:36.574735 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.575235 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.575237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.575238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.575307 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.575609 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.575611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.575612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.575764 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:36.575839 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:36.576207 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:46:36.576300 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":364}
I20260928 02:46:36.576581 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.568339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.28.248.129:60109)
0928 02:46:36.568679 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:36.568694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:36.568772 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:36.569300 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:36.569303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:36.569507 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:36.569615 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.569621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.570766 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.570772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.571519 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.571528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.571960 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.571963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.572059 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.572661 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:36.572680 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:36.573131 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:36.574848 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.574853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.574859 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.575136 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.575400 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.575405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.575409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.575517 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.575909 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:36.575914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:36.576027 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 02:46:36.576160 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:46:36.576307 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for oryx/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab.
Entry for principal oryx/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab.
I20260928 02:46:36.595971 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37681
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:36.707264 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:36.707566 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:36.707613 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:36.711897 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:36.711977 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:36.712018 4952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:36.712057 4952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:36.717391 4952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:36.718446 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:36.719487 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:36.729470 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.729518 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.729496 4960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:36.729810 4952 server_base.cc:1034] running on GCE node
I20260928 02:46:36.730252 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:36.730983 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:36.732195 4952 hybrid_clock.cc:648] HybridClock initialized: now 1790563596732169 us; error 50 us; skew 500 ppm
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:36.735155 4952 init.cc:377] Logged in from keytab as oryx/127.28.248.130@KRBTEST.COM (short username oryx)
I20260928 02:46:36.736016 4952 webserver.cc:460] Webserver started at http://127.28.248.130:34865/ using document root <none> and password file <none>
I20260928 02:46:36.736583 4952 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:36.736704 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:36.736974 4952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:36.739069 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "26d910e3d7604bbca93ea792c3a9cc1f"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "956fe85067e7e087b5e1a23b2f9eace2"
server_key_iv: "7f6b740a8a019d5e0a4ed9b2d8a59aea"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.739642 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "26d910e3d7604bbca93ea792c3a9cc1f"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "956fe85067e7e087b5e1a23b2f9eace2"
server_key_iv: "7f6b740a8a019d5e0a4ed9b2d8a59aea"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.743438 4952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:46:36.746042 4967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.747334 4952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:36.747453 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "26d910e3d7604bbca93ea792c3a9cc1f"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "956fe85067e7e087b5e1a23b2f9eace2"
server_key_iv: "7f6b740a8a019d5e0a4ed9b2d8a59aea"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.747566 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:36.761170 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:36.764477 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:36.764730 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:36.766129 4952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:36.766175 4952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.766242 4952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:36.766273 4952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:36.777386 4952 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:42131
I20260928 02:46:36.777592 5082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:42131 every 8 connection(s)
I20260928 02:46:36.778312 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:36.779954 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.767080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:58055 (local address 127.28.248.190:37681)
0928 02:46:36.767211 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:36.767215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:36.768293 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 02:46:36.769410 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:36.769417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:36.769488 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:36.769602 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:36.771115 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.772019 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.773563 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.773800 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.773916 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.776355 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:36.776358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:36.776360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:36.776385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:36.777947 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.778432 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.778437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.778439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.778528 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.778918 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.778921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.778923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.779181 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:36.779298 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:36.779739 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 02:46:36.779838 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:36.780831 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.767626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.28.248.130:58055)
0928 02:46:36.768142 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:36.768185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:36.769167 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:36.769800 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:36.769810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:36.770262 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:36.770958 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.770979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.772161 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.772165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.773421 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.773430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.773963 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.773968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.774149 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.774717 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:36.774733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:36.776160 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:36.778064 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.778069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.778091 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.778328 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.778635 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.778639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.778640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.778808 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.779300 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:36.779305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:36.779532 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 02:46:36.779687 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:46:36.779922 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":141,"threads_started":1}
I20260928 02:46:36.782009 5083 heartbeater.cc:347] Connected to a master server at 127.28.248.190:37681
I20260928 02:46:36.782142 5083 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:36.782291 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 4952
I20260928 02:46:36.782426 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:36.782706 29666 external_mini_cluster.cc:1428] Setting key bf45c27a4dcdcaad9fcb881105b486c8
I20260928 02:46:36.782680 5083 heartbeater.cc:511] Master 127.28.248.190:37681 requested a full tablet report, sending...
I20260928 02:46:36.783748 4739 ts_manager.cc:194] Registered new tserver with Master: 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131)
I20260928 02:46:36.784505 4739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:58055
WARNING: no policy specified for oryx/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:36.793562 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.786113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:51311 (local address 127.28.248.190:37681)
0928 02:46:36.786259 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:36.786265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:36.786281 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:46:36.786323 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:36.786327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:36.786375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:36.786463 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:36.786957 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.787561 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.788887 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.789096 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.789173 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.790492 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:36.790496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:36.790498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:36.790529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:36.791923 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.792398 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.792400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.792401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.792473 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:36.792858 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:36.792860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:36.792861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:36.793079 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:36.793160 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:36.793359 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:46:36.793448 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:46:36.793633 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.784274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.28.248.130:51311)
0928 02:46:36.784569 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:36.784584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:36.784688 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:36.786514 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:36.786517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:36.786728 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:36.786838 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.786844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.787703 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.787709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.788772 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.788783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.789215 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.789221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.789353 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.789853 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:36.789874 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:36.790358 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:36.792037 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.792043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.792046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.792303 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.792645 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:36.792651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:36.792653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:36.792748 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:36.793169 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:36.793171 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:36.793245 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 02:46:36.793319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:46:36.793422 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab.
Entry for principal oryx/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab.
I20260928 02:46:36.809835 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:37681
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:36.923497 5092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:36.923745 5092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:36.923787 5092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:36.927909 5092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:36.927971 5092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:36.928009 5092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:46:36.928041 5092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:36.933202 5092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.248.190:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:36.934150 5092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:36.935102 5092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:36.944588 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.944566 5098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:36.944561 5097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:36.944808 5092 server_base.cc:1034] running on GCE node
I20260928 02:46:36.945355 5092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:36.946031 5092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:36.947209 5092 hybrid_clock.cc:648] HybridClock initialized: now 1790563596947193 us; error 40 us; skew 500 ppm
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:36.949882 5092 init.cc:377] Logged in from keytab as oryx/127.28.248.131@KRBTEST.COM (short username oryx)
I20260928 02:46:36.950866 5092 webserver.cc:460] Webserver started at http://127.28.248.131:45855/ using document root <none> and password file <none>
I20260928 02:46:36.951395 5092 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:36.951496 5092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:36.951666 5092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:36.953745 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "880e837f93ee30b899eab25e89419c5d"
server_key_iv: "64bf74f28d0915d866d2d9561068e383"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.954291 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "880e837f93ee30b899eab25e89419c5d"
server_key_iv: "64bf74f28d0915d866d2d9561068e383"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.957938 5092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:46:36.960300 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.961488 5092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:36.961603 5092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
format_stamp: "Formatted at 2026-09-28 02:46:36 on dist-test-slave-r238"
server_key: "880e837f93ee30b899eab25e89419c5d"
server_key_iv: "64bf74f28d0915d866d2d9561068e383"
server_key_version: "encryptionkey@0"
I20260928 02:46:36.961702 5092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:36.983608 5092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:36.986573 5092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:36.986768 5092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:36.988217 5092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:36.988266 5092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:36.988305 5092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:36.988358 5092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:46:36 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:36.999778 5092 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:41603
I20260928 02:46:36.999855 5222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:41603 every 8 connection(s)
I20260928 02:46:37.000912 5092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:37.002445 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.989025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:58233 (local address 127.28.248.190:37681)
0928 02:46:36.989157 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:36.989161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:36.990231 (+ 1070us) server_negotiation.cc:407] Connection header received
0928 02:46:36.991193 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:36.991196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:36.991254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:36.991336 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:36.992655 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.993555 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.994762 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.995166 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.995297 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.998389 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:36.998395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:36.998399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:36.998432 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:37.000142 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.000657 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.000660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.000662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.000754 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.001214 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.001220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.001224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.001477 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:37.001575 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.002223 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.002329 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:37.003263 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:36.989584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.28.248.131:58233)
0928 02:46:36.990077 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:36.990119 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:36.990975 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:36.991679 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:36.991686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:36.992020 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:36.992470 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.992484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.993714 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.993719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.994658 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:36.994668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:36.995357 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:36.995363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:36.995570 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:36.996223 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:36.996246 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:36.998203 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:37.000261 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:37.000265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:37.000276 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:37.000529 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:37.000884 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:37.000887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:37.000889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:37.001111 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:37.001632 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:37.001640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:37.001966 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.002163 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.002409 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260928 02:46:37.004570 5223 heartbeater.cc:347] Connected to a master server at 127.28.248.190:37681
I20260928 02:46:37.004774 5223 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:37.005298 5223 heartbeater.cc:511] Master 127.28.248.190:37681 requested a full tablet report, sending...
I20260928 02:46:37.006347 4740 ts_manager.cc:194] Registered new tserver with Master: 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.006621 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5092
I20260928 02:46:37.006726 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:37.007027 29666 external_mini_cluster.cc:1428] Setting key a224a955b9c41a92b3c09874a36bb677
I20260928 02:46:37.007134 4740 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:58233
I20260928 02:46:37.009021 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:37.015090 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.007385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:49549 (local address 127.28.248.190:37681)
0928 02:46:37.007503 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.007507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.007520 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:37.007555 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.007559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.007604 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.007698 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:37.008299 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.009088 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.009943 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.010391 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.010502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.011779 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:37.011782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:37.011784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:37.011805 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:37.013554 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.013940 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.013946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.013951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.014055 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.014321 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.014326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.014330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.014566 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:37.014670 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.014877 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.014978 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:46:37.015198 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.006653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.28.248.131:49549)
0928 02:46:37.006940 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.006955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.007036 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.007814 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.007817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.008034 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:37.008173 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.008181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.009238 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.009241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.009825 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.009833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.010505 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.010508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.010586 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.011129 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:37.011146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:37.011634 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:37.013665 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:37.013667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:37.013669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:37.013859 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:37.014158 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:37.014160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:37.014161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:37.014232 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:37.014660 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:37.014662 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:37.014729 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.014804 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.014963 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":109,"threads_started":1}
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:46:37.034613 4789 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:37.028387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.28.248.190:37681)
0928 02:46:37.028544 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.028547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.028560 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:37.028643 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.028646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.028696 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.028782 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:37.029605 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.030197 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.031029 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.031312 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.031394 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.034480 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47854: BlockingRecv error: recv got EOF from 127.0.0.1:47854 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:37.043865 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.036099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.28.248.190:37681)
0928 02:46:37.036232 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.036236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.036355 (+ 119us) server_negotiation.cc:407] Connection header received
0928 02:46:37.036512 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.036515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.036550 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.036663 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:37.037232 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.037747 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.038530 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.038738 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.038844 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.041033 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:37.041036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:37.041038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:37.041059 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:37.042213 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.042633 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.042637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.042639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.042727 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.043002 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.043006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.043007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.043199 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:37.043337 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.043565 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.043676 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:46:37.046838 4740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:37.049831 4740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:37.064339 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.059110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.28.248.131:41603)
0928 02:46:37.059554 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.059565 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.059785 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:46:37.059950 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.059954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.060136 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.060255 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.060650 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.061726 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063172 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.063691 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063844 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.063994 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.064094 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.064184 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":124,"threads_started":1}
I20260928 02:46:37.064330 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.058506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41603 (local address 127.0.0.1:50850)
0928 02:46:37.059678 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.059689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.059805 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.060406 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.060409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.060420 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.060499 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.060504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.061841 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.061844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.063040 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.063048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.063814 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.063919 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.063953 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.064069 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.064184 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":73,"spinlock_wait_cycles":1792,"thread_start_us":96,"threads_started":1}
I20260928 02:46:37.064738 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.058321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:42131 (local address 127.0.0.1:51498)
0928 02:46:37.058957 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.059019 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.059144 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.059954 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.059959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.059980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.060112 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.060118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.061433 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.061436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.063010 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.063021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.064137 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.064141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.064324 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.064383 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.064454 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.064588 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":118,"thread_start_us":77,"threads_started":1}
I20260928 02:46:37.065029 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.059137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.28.248.129:42331)
0928 02:46:37.059566 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.059583 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.059849 (+ 266us) server_negotiation.cc:407] Connection header received
0928 02:46:37.059930 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.059935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.060136 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.060268 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.060663 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.061423 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063012 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.063810 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063950 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.064070 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.064549 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.064665 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":86,"threads_started":1}
I20260928 02:46:37.065641 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.058716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.0.0.1:55390)
0928 02:46:37.059692 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.059706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.059792 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.060388 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.060392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.060409 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.060499 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.060504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.061539 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.061542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.062896 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.062905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.064137 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.064141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.064303 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.064382 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.064454 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.065484 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":1308,"thread_start_us":85,"threads_started":1}
I20260928 02:46:37.065801 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.058482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.28.248.130:42131)
0928 02:46:37.059431 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.059443 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.059465 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:46:37.059540 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.059546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.059766 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.059954 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.060234 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.061312 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063119 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.063826 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.063950 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.064070 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.065030 (+ 960us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.065129 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":107,"threads_started":1}
I20260928 02:46:37.067693 5157 tablet_service.cc:1467] Processing CreateTablet for tablet f25eb77b78714925b1a282cff8857628 (DEFAULT_TABLE table=test-table [id=1ac92fb37eb8482abea9a8d11afc655d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.067996 5017 tablet_service.cc:1467] Processing CreateTablet for tablet f25eb77b78714925b1a282cff8857628 (DEFAULT_TABLE table=test-table [id=1ac92fb37eb8482abea9a8d11afc655d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.068784 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25eb77b78714925b1a282cff8857628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.069028 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25eb77b78714925b1a282cff8857628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.075498 5251 tablet_bootstrap.cc:492] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Bootstrap starting.
I20260928 02:46:37.075382 4866 tablet_service.cc:1467] Processing CreateTablet for tablet f25eb77b78714925b1a282cff8857628 (DEFAULT_TABLE table=test-table [id=1ac92fb37eb8482abea9a8d11afc655d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.076179 5252 tablet_bootstrap.cc:492] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Bootstrap starting.
I20260928 02:46:37.076402 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25eb77b78714925b1a282cff8857628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.078078 5252 tablet_bootstrap.cc:654] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.078078 5251 tablet_bootstrap.cc:654] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.078934 5252 log.cc:824] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:37.078934 5251 log.cc:824] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:37.080907 5252 tablet_bootstrap.cc:492] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: No bootstrap required, opened a new log
I20260928 02:46:37.080907 5251 tablet_bootstrap.cc:492] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: No bootstrap required, opened a new log
I20260928 02:46:37.081097 5252 ts_tablet_manager.cc:1397] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:46:37.081097 5251 ts_tablet_manager.cc:1397] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:46:37.082782 5255 tablet_bootstrap.cc:492] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Bootstrap starting.
I20260928 02:46:37.084723 5255 tablet_bootstrap.cc:654] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.085032 5252 raft_consensus.cc:348] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.085032 5251 raft_consensus.cc:348] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.085291 5251 raft_consensus.cc:374] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.085291 5252 raft_consensus.cc:374] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.085350 5252 raft_consensus.cc:729] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19496ebbb86243eab7ba76d2d8faa7ce, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.085358 5251 raft_consensus.cc:729] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.085496 5255 log.cc:824] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:37.085796 5251 consensus_queue.cc:260] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.085796 5252 consensus_queue.cc:260] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.086680 5083 heartbeater.cc:503] Master 127.28.248.190:37681 was elected leader, sending a full tablet report...
I20260928 02:46:37.086716 5223 heartbeater.cc:503] Master 127.28.248.190:37681 was elected leader, sending a full tablet report...
I20260928 02:46:37.086784 5251 ts_tablet_manager.cc:1428] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:46:37.087891 5252 ts_tablet_manager.cc:1428] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:46:37.088335 5255 tablet_bootstrap.cc:492] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: No bootstrap required, opened a new log
I20260928 02:46:37.088521 5255 ts_tablet_manager.cc:1397] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 02:46:37.093137 5255 raft_consensus.cc:348] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.093345 5255 raft_consensus.cc:374] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.093420 5255 raft_consensus.cc:729] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.093806 5255 consensus_queue.cc:260] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.094421 4943 heartbeater.cc:503] Master 127.28.248.190:37681 was elected leader, sending a full tablet report...
I20260928 02:46:37.094755 5255 ts_tablet_manager.cc:1428] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:46:37.169399 5259 raft_consensus.cc:482] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:37.169598 5259 raft_consensus.cc:504] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.170764 5259 leader_election.cc:288] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.176221 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.171168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:46723 (local address 127.28.248.131:41603)
0928 02:46:37.171307 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.171314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.171541 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:46:37.171618 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.171621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.171671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.171757 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.172141 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.172975 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.174851 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.175483 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.175616 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.175667 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.175991 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.176080 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:46:37.176331 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.171020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41603 (local address 127.28.248.129:46723)
0928 02:46:37.171429 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.171461 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.171546 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.171785 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.171788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.171830 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.172019 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.172027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.173128 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.173132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.174720 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.174732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.175647 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.175651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.175815 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.175849 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.175969 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.176094 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":231,"spinlock_wait_cycles":44416,"thread_start_us":115,"threads_started":1}
I20260928 02:46:37.176792 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.171163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:40955 (local address 127.28.248.130:42131)
0928 02:46:37.171377 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.171380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.171454 (+ 74us) server_negotiation.cc:407] Connection header received
0928 02:46:37.171618 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.171621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.171671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.171757 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.172141 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.172975 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.174924 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.175512 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.175616 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.175667 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.176565 (+ 898us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.176667 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 02:46:37.177068 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25eb77b78714925b1a282cff8857628" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" is_pre_election: true
I20260928 02:46:37.177241 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.171011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:42131 (local address 127.28.248.129:40955)
0928 02:46:37.171351 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.171366 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.171496 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.171785 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.171788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.171828 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.172019 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.172027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.173128 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.173132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.174719 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.174732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.175647 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.175651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.175857 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.175881 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.175963 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.176094 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":255,"spinlock_wait_cycles":30720,"thread_start_us":127,"threads_started":1}
I20260928 02:46:37.177456 5170 raft_consensus.cc:2454] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 0.
I20260928 02:46:37.177884 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25eb77b78714925b1a282cff8857628" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" is_pre_election: true
I20260928 02:46:37.178073 4830 leader_election.cc:302] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19496ebbb86243eab7ba76d2d8faa7ce, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.178200 5037 raft_consensus.cc:2454] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 0.
I20260928 02:46:37.178386 5259 raft_consensus.cc:2781] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:37.178475 5259 raft_consensus.cc:482] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:37.178565 5259 raft_consensus.cc:3037] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.179641 5259 raft_consensus.cc:504] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.179993 5259 leader_election.cc:288] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 election: Requested vote from peers 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.180307 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25eb77b78714925b1a282cff8857628" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d910e3d7604bbca93ea792c3a9cc1f"
I20260928 02:46:37.180419 5037 raft_consensus.cc:3037] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.180403 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25eb77b78714925b1a282cff8857628" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
I20260928 02:46:37.180527 5170 raft_consensus.cc:3037] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.181430 5037 raft_consensus.cc:2454] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 1.
I20260928 02:46:37.181699 5170 raft_consensus.cc:2454] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 1.
I20260928 02:46:37.181900 4829 leader_election.cc:302] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d910e3d7604bbca93ea792c3a9cc1f, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.182057 5259 raft_consensus.cc:2781] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:37.182267 5259 raft_consensus.cc:686] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 1 LEADER]: Becoming Leader. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Running, Role: LEADER
I20260928 02:46:37.182613 5259 consensus_queue.cc:237] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } }
I20260928 02:46:37.185737 4739 catalog_manager.cc:5495] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f8468eb794442c92d916a922f8f584 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } health_report { overall_health: HEALTHY } } }
I20260928 02:46:37.214416 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.211264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.28.248.129:42331)
0928 02:46:37.211414 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.211418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.211433 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:37.211506 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.211508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.211562 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.211653 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:37.211856 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.212356 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.213156 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.213380 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.213457 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.213680 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.214094 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.214178 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.214261 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.214308 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:46:37.220500 5170 raft_consensus.cc:1261] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Refusing update from remote peer e2f8468eb794442c92d916a922f8f584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.220525 5037 raft_consensus.cc:1261] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Refusing update from remote peer e2f8468eb794442c92d916a922f8f584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.221242 5259 consensus_queue.cc:1035] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.221433 5262 consensus_queue.cc:1035] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.227384 5267 mvcc.cc:205] Tried to move back new op lower bound from 7334148494207414272 to 7334148494062780416. Current Snapshot: MvccSnapshot[applied={T|T < 7334148494207414272}]
I20260928 02:46:37.228178 5265 mvcc.cc:205] Tried to move back new op lower bound from 7334148494207414272 to 7334148494062780416. Current Snapshot: MvccSnapshot[applied={T|T < 7334148494207414272}]
I20260928 02:46:37.236236 4738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:37.236413 4738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:37.238679 4738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Sending DeleteTablet for 3 replicas of tablet f25eb77b78714925b1a282cff8857628
I20260928 02:46:37.239449 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet f25eb77b78714925b1a282cff8857628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51498
I20260928 02:46:37.239449 5157 tablet_service.cc:1514] Processing DeleteTablet for tablet f25eb77b78714925b1a282cff8857628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:50850
I20260928 02:46:37.239581 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet f25eb77b78714925b1a282cff8857628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55390
I20260928 02:46:37.240005 5276 tablet_replica.cc:330] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: stopping tablet replica
I20260928 02:46:37.240243 5275 tablet_replica.cc:330] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: stopping tablet replica
I20260928 02:46:37.240319 5276 raft_consensus.cc:2229] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:37.240509 5275 raft_consensus.cc:2229] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:37.240731 5276 raft_consensus.cc:2258] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.240810 5275 raft_consensus.cc:2258] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.240893 4738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:46:37.241154 5274 tablet_replica.cc:330] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: stopping tablet replica
W20260928 02:46:37.241276 4738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:37.241407 5274 raft_consensus.cc:2229] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:37.241657 5274 raft_consensus.cc:2258] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.243512 5275 ts_tablet_manager.cc:1905] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.244398 5276 ts_tablet_manager.cc:1905] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.245672 4866 tablet_service.cc:1467] Processing CreateTablet for tablet c87b4f88c6e542d0ab328917709cdc91 (DEFAULT_TABLE table=test-table [id=bd40602252fa4f6f90471d2131d387cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.245911 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87b4f88c6e542d0ab328917709cdc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.245950 5157 tablet_service.cc:1467] Processing CreateTablet for tablet c87b4f88c6e542d0ab328917709cdc91 (DEFAULT_TABLE table=test-table [id=bd40602252fa4f6f90471d2131d387cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.246042 5017 tablet_service.cc:1467] Processing CreateTablet for tablet c87b4f88c6e542d0ab328917709cdc91 (DEFAULT_TABLE table=test-table [id=bd40602252fa4f6f90471d2131d387cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.246196 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87b4f88c6e542d0ab328917709cdc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.246264 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87b4f88c6e542d0ab328917709cdc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.248274 5255 tablet_bootstrap.cc:492] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Bootstrap starting.
I20260928 02:46:37.248484 5252 tablet_bootstrap.cc:492] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Bootstrap starting.
I20260928 02:46:37.249207 5255 tablet_bootstrap.cc:654] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.249490 5252 tablet_bootstrap.cc:654] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.249804 5251 tablet_bootstrap.cc:492] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Bootstrap starting.
I20260928 02:46:37.250792 5251 tablet_bootstrap.cc:654] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.252799 5276 ts_tablet_manager.cc:1918] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:37.252911 5276 log.cc:1196] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/f25eb77b78714925b1a282cff8857628
I20260928 02:46:37.252985 5275 ts_tablet_manager.cc:1918] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:37.253079 5275 log.cc:1196] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/f25eb77b78714925b1a282cff8857628
I20260928 02:46:37.253243 5276 ts_tablet_manager.cc:1939] T f25eb77b78714925b1a282cff8857628 P e2f8468eb794442c92d916a922f8f584: Deleting consensus metadata
I20260928 02:46:37.253400 5275 ts_tablet_manager.cc:1939] T f25eb77b78714925b1a282cff8857628 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting consensus metadata
I20260928 02:46:37.254323 4724 catalog_manager.cc:4841] TS e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331): tablet f25eb77b78714925b1a282cff8857628 (table test-table [id=1ac92fb37eb8482abea9a8d11afc655d]) successfully deleted
I20260928 02:46:37.254472 4726 catalog_manager.cc:4841] TS 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603): tablet f25eb77b78714925b1a282cff8857628 (table test-table [id=1ac92fb37eb8482abea9a8d11afc655d]) successfully deleted
I20260928 02:46:37.255568 5251 tablet_bootstrap.cc:492] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: No bootstrap required, opened a new log
I20260928 02:46:37.255668 5251 ts_tablet_manager.cc:1397] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:46:37.255730 5255 tablet_bootstrap.cc:492] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: No bootstrap required, opened a new log
I20260928 02:46:37.255798 5255 ts_tablet_manager.cc:1397] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 02:46:37.256102 5251 raft_consensus.cc:348] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.256203 5251 raft_consensus.cc:374] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.256233 5251 raft_consensus.cc:729] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.256218 5255 raft_consensus.cc:348] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.256283 5274 ts_tablet_manager.cc:1905] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.256321 5255 raft_consensus.cc:374] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.256352 5255 raft_consensus.cc:729] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.256350 5251 consensus_queue.cc:260] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.256459 5255 consensus_queue.cc:260] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.256640 5251 ts_tablet_manager.cc:1428] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.256705 5255 ts_tablet_manager.cc:1428] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.259112 5274 ts_tablet_manager.cc:1918] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:37.259209 5274 log.cc:1196] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/f25eb77b78714925b1a282cff8857628
I20260928 02:46:37.259514 5274 ts_tablet_manager.cc:1939] T f25eb77b78714925b1a282cff8857628 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting consensus metadata
I20260928 02:46:37.260421 4725 catalog_manager.cc:4841] TS 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131): tablet f25eb77b78714925b1a282cff8857628 (table test-table [id=1ac92fb37eb8482abea9a8d11afc655d]) successfully deleted
I20260928 02:46:37.261240 5252 tablet_bootstrap.cc:492] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: No bootstrap required, opened a new log
I20260928 02:46:37.261350 5252 ts_tablet_manager.cc:1397] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 02:46:37.261735 5252 raft_consensus.cc:348] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.261852 5252 raft_consensus.cc:374] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.261896 5252 raft_consensus.cc:729] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19496ebbb86243eab7ba76d2d8faa7ce, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.262024 5252 consensus_queue.cc:260] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.262291 5252 ts_tablet_manager.cc:1428] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 02:46:37.279520 5084 tablet.cc:2369] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:46:37.312863 4944 tablet.cc:2369] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:37.436546 5258 raft_consensus.cc:482] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:37.436787 5258 raft_consensus.cc:504] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.437914 5258 leader_election.cc:288] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.441931 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.438244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:56097 (local address 127.28.248.131:41603)
0928 02:46:37.438372 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.438379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.438559 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:46:37.438667 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.438671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.438721 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.438800 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.439117 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.439666 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.440933 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.441316 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.441447 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.441499 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.441751 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.441823 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:46:37.441900 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.438273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:38965 (local address 127.28.248.129:42331)
0928 02:46:37.438471 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.438477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.438676 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:46:37.438805 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.438809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.438859 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.438934 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:37.439216 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.439694 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.440909 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.441308 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.441437 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.441508 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.441699 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.441772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 02:46:37.441891 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.438162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.28.248.130:38965)
0928 02:46:37.438578 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.438593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.438691 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.438964 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.438966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.438979 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.439108 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.439114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.439831 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.439834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.440791 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.440799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.441425 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.441429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.441543 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.441587 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.441646 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.441733 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":65,"thread_start_us":73,"threads_started":1}
I20260928 02:46:37.441912 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.438155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41603 (local address 127.28.248.130:56097)
0928 02:46:37.438452 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.438468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.438577 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.438826 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.438829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.438844 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:37.438995 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.439005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.439780 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.439787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.440795 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.440804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.441455 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.441459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.441558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.441616 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.441713 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.441778 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":77,"thread_start_us":86,"threads_started":1}
I20260928 02:46:37.442600 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87b4f88c6e542d0ab328917709cdc91" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" is_pre_election: true
I20260928 02:46:37.442608 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87b4f88c6e542d0ab328917709cdc91" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f8468eb794442c92d916a922f8f584" is_pre_election: true
I20260928 02:46:37.442755 5170 raft_consensus.cc:2454] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 0.
I20260928 02:46:37.442840 4879 raft_consensus.cc:2454] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 0.
I20260928 02:46:37.443150 4970 leader_election.cc:302] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19496ebbb86243eab7ba76d2d8faa7ce, 26d910e3d7604bbca93ea792c3a9cc1f; no voters:
I20260928 02:46:37.443429 5258 raft_consensus.cc:2781] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:37.443517 5258 raft_consensus.cc:482] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:37.443558 5258 raft_consensus.cc:3037] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.444485 5258 raft_consensus.cc:504] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.444887 5258 leader_election.cc:288] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 election: Requested vote from peers e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.445204 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87b4f88c6e542d0ab328917709cdc91" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f8468eb794442c92d916a922f8f584"
I20260928 02:46:37.445230 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87b4f88c6e542d0ab328917709cdc91" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
I20260928 02:46:37.445323 5170 raft_consensus.cc:3037] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.445323 4879 raft_consensus.cc:3037] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.446394 4879 raft_consensus.cc:2454] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 1.
I20260928 02:46:37.446394 5170 raft_consensus.cc:2454] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 1.
I20260928 02:46:37.446669 4968 leader_election.cc:302] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d910e3d7604bbca93ea792c3a9cc1f, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.446856 5258 raft_consensus.cc:2781] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:37.447031 5258 raft_consensus.cc:686] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 LEADER]: Becoming Leader. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Running, Role: LEADER
I20260928 02:46:37.447286 5258 consensus_queue.cc:237] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.449290 4739 catalog_manager.cc:5495] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130). New cstate: current_term: 1 leader_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } health_report { overall_health: UNKNOWN } } }
W20260928 02:46:37.502188 5224 tablet.cc:2369] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.190@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:37.518720 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.510472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:37681 (local address 127.0.0.1:47876)
0928 02:46:37.510779 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.510792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.510928 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.511172 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.511175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.511652 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:37.511750 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.511756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.512526 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.512532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.513176 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.513184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.513617 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.513620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.513697 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.514224 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:37.514236 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:37.515647 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:37.517221 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:37.517229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:37.517234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:37.517467 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:37.517708 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:37.517709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:37.517711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:37.517787 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:37.518203 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:37.518208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:37.518315 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.518404 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.518540 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260928 02:46:37.518723 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.510479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.28.248.190:37681)
0928 02:46:37.510588 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.510592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.510869 (+ 277us) server_negotiation.cc:407] Connection header received
0928 02:46:37.511028 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.511031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.511073 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.511157 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:37.511871 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.512416 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.513291 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.513506 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.513613 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.515778 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:37.515781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:37.515783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:37.515804 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:37.517123 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.517559 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.517561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.517562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.517630 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.517887 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.517891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.517893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.518113 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:37.518232 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.518463 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.518597 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":31}
I20260928 02:46:37.522521 4739 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:47876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:46:37.523290 4739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:37.528035 4866 tablet_service.cc:1467] Processing CreateTablet for tablet f12ec90b82e14a96bdfe8e101978aecb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c340cff176544319e04a821b2dae1f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:37.528035 5017 tablet_service.cc:1467] Processing CreateTablet for tablet f12ec90b82e14a96bdfe8e101978aecb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c340cff176544319e04a821b2dae1f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:37.528213 5157 tablet_service.cc:1467] Processing CreateTablet for tablet f12ec90b82e14a96bdfe8e101978aecb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2c340cff176544319e04a821b2dae1f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:46:37.528420 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f12ec90b82e14a96bdfe8e101978aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.528420 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f12ec90b82e14a96bdfe8e101978aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.528641 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f12ec90b82e14a96bdfe8e101978aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.531219 5255 tablet_bootstrap.cc:492] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: Bootstrap starting.
I20260928 02:46:37.531261 5251 tablet_bootstrap.cc:492] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: Bootstrap starting.
I20260928 02:46:37.531307 5252 tablet_bootstrap.cc:492] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: Bootstrap starting.
I20260928 02:46:37.532012 5255 tablet_bootstrap.cc:654] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.532176 5251 tablet_bootstrap.cc:654] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.532176 5252 tablet_bootstrap.cc:654] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.533205 5255 tablet_bootstrap.cc:492] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: No bootstrap required, opened a new log
I20260928 02:46:37.533296 5255 ts_tablet_manager.cc:1397] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:37.533340 5251 tablet_bootstrap.cc:492] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: No bootstrap required, opened a new log
I20260928 02:46:37.533427 5251 ts_tablet_manager.cc:1397] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:37.533488 5252 tablet_bootstrap.cc:492] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: No bootstrap required, opened a new log
I20260928 02:46:37.533571 5252 ts_tablet_manager.cc:1397] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:37.533712 5255 raft_consensus.cc:348] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.533826 5255 raft_consensus.cc:374] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.533859 5255 raft_consensus.cc:729] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.533877 5251 raft_consensus.cc:348] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.533993 5251 raft_consensus.cc:374] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.534035 5251 raft_consensus.cc:729] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.533991 5255 consensus_queue.cc:260] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.534001 5252 raft_consensus.cc:348] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.534088 5252 raft_consensus.cc:374] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.534123 5252 raft_consensus.cc:729] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19496ebbb86243eab7ba76d2d8faa7ce, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.534165 5251 consensus_queue.cc:260] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.534272 5255 ts_tablet_manager.cc:1428] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.534236 5252 consensus_queue.cc:260] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.534435 5251 ts_tablet_manager.cc:1428] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:37.534452 5259 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.534541 5252 ts_tablet_manager.cc:1428] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.534570 5259 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.534478 5257 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.534621 5257 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.534720 5258 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.534827 5258 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.563159 5259 raft_consensus.cc:482] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:37.563268 5259 raft_consensus.cc:504] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.563699 5259 leader_election.cc:288] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.564015 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f12ec90b82e14a96bdfe8e101978aecb" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" is_pre_election: true
I20260928 02:46:37.564196 5037 raft_consensus.cc:2454] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 0.
I20260928 02:46:37.564180 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f12ec90b82e14a96bdfe8e101978aecb" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" is_pre_election: true
I20260928 02:46:37.564320 5170 raft_consensus.cc:2454] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 0.
I20260928 02:46:37.564518 4829 leader_election.cc:302] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d910e3d7604bbca93ea792c3a9cc1f, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.564697 5259 raft_consensus.cc:2781] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:37.564772 5259 raft_consensus.cc:482] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:37.564813 5259 raft_consensus.cc:3037] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.565871 5259 raft_consensus.cc:504] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.566211 5259 leader_election.cc:288] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 election: Requested vote from peers 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.566529 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f12ec90b82e14a96bdfe8e101978aecb" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
I20260928 02:46:37.566555 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f12ec90b82e14a96bdfe8e101978aecb" candidate_uuid: "e2f8468eb794442c92d916a922f8f584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d910e3d7604bbca93ea792c3a9cc1f"
I20260928 02:46:37.566663 5170 raft_consensus.cc:3037] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.566663 5037 raft_consensus.cc:3037] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.567683 5037 raft_consensus.cc:2454] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 1.
I20260928 02:46:37.567730 5170 raft_consensus.cc:2454] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f8468eb794442c92d916a922f8f584 in term 1.
I20260928 02:46:37.567956 4830 leader_election.cc:302] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19496ebbb86243eab7ba76d2d8faa7ce, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.568109 5259 raft_consensus.cc:2781] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:37.568229 5259 raft_consensus.cc:686] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [term 1 LEADER]: Becoming Leader. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Running, Role: LEADER
I20260928 02:46:37.568369 5259 consensus_queue.cc:237] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.569047 5262 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: TxnStatusTablet state changed. Reason: New leader e2f8468eb794442c92d916a922f8f584. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.569154 5262 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:37.569399 5294 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:46:37.569525 5294 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:46:37.570006 4739 catalog_manager.cc:5495] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f8468eb794442c92d916a922f8f584 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:37.591241 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.587754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.28.248.129:42331)
0928 02:46:37.587873 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.587877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.587979 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:46:37.588150 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.588152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.588193 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.588249 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:37.588522 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.589094 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.590274 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.590473 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.590563 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.590851 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.590913 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.590964 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.591092 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.591139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:46:37.591282 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.587666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.0.0.1:55412)
0928 02:46:37.587897 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.587912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.588029 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.588292 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.588295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.588303 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:37.588413 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.588419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.589194 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.589198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.590160 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.590170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.590586 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.590590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.590679 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.590729 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.590995 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.591007 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.591073 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.591150 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 02:46:37.593684 5037 raft_consensus.cc:1261] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Refusing update from remote peer e2f8468eb794442c92d916a922f8f584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.593698 5170 raft_consensus.cc:1261] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Refusing update from remote peer e2f8468eb794442c92d916a922f8f584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.594117 5259 consensus_queue.cc:1035] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.594269 5262 consensus_queue.cc:1035] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.595568 5258 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: TxnStatusTablet state changed. Reason: New leader e2f8468eb794442c92d916a922f8f584. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.595594 5257 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: TxnStatusTablet state changed. Reason: New leader e2f8468eb794442c92d916a922f8f584. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.595672 5258 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.595697 5257 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.597772 5259 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.597870 5259 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:37.597978 5259 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.598042 5259 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P e2f8468eb794442c92d916a922f8f584: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:46:37.598488 5258 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.598585 5258 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P 26d910e3d7604bbca93ea792c3a9cc1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.599115 5266 mvcc.cc:205] Tried to move back new op lower bound from 7334148495740026880 to 7334148495641649152. Current Snapshot: MvccSnapshot[applied={T|T < 7334148495740026880}]
I20260928 02:46:37.599298 5257 tablet_replica.cc:439] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2f8468eb794442c92d916a922f8f584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } } }
I20260928 02:46:37.599386 5257 tablet_replica.cc:442] T f12ec90b82e14a96bdfe8e101978aecb P 19496ebbb86243eab7ba76d2d8faa7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:46:37.606026 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.602287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.28.248.130:42131)
0928 02:46:37.602584 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.602588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.602597 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:46:37.602653 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.602655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.602697 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.602807 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:37.603037 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.603539 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.604378 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.604766 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.604859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.605127 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.605557 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.605684 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.605845 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.605896 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260928 02:46:37.616245 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.612974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:50089 (local address 127.28.248.129:42331)
0928 02:46:37.613111 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.613117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.613249 (+ 132us) server_negotiation.cc:407] Connection header received
0928 02:46:37.613433 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.613436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.613493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.613583 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:37.613824 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.614313 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.615184 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.615431 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.615540 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.615813 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.615891 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.615969 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.616057 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.616107 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:46:37.616546 5301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.612857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.28.248.130:50089)
0928 02:46:37.613153 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.613171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.613299 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.613594 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.613597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.613605 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:37.613719 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.613725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.614424 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.614431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.615065 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.615073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.615545 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.615549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.615643 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.615683 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.615975 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.615983 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.616049 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.616122 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260928 02:46:37.623804 5170 raft_consensus.cc:1261] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Refusing update from remote peer 26d910e3d7604bbca93ea792c3a9cc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.623991 4879 raft_consensus.cc:1261] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Refusing update from remote peer 26d910e3d7604bbca93ea792c3a9cc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.624358 5282 consensus_queue.cc:1035] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.624522 5258 consensus_queue.cc:1035] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.644933 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.641283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:60751 (local address 127.28.248.130:42131)
0928 02:46:37.641384 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.641389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.641433 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:46:37.641627 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.641631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.641686 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.641734 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:37.642170 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.642839 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.643689 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.644005 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.644120 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.644408 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.644490 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.644537 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.644754 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.644817 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:46:37.645329 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.641028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:42131 (local address 127.28.248.129:60751)
0928 02:46:37.641353 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:37.641370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:37.641510 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:37.641913 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:37.641915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:37.641925 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:37.642042 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.642049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.642959 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.642961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.643570 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:37.643576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.644118 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.644122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:37.644204 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.644258 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.644644 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.644651 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:46:37.645141 (+ 490us) client_negotiation.cc:240] Negotiation successful
0928 02:46:37.645204 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
I20260928 02:46:37.672654 4738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:37.672828 4738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:37.675143 4738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Sending DeleteTablet for 3 replicas of tablet c87b4f88c6e542d0ab328917709cdc91
I20260928 02:46:37.675678 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet c87b4f88c6e542d0ab328917709cdc91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55390
I20260928 02:46:37.675705 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet c87b4f88c6e542d0ab328917709cdc91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51498
I20260928 02:46:37.675863 5276 tablet_replica.cc:330] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: stopping tablet replica
I20260928 02:46:37.675865 5274 tablet_replica.cc:330] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: stopping tablet replica
I20260928 02:46:37.675979 5276 raft_consensus.cc:2229] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:37.675984 5274 raft_consensus.cc:2229] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:37.676090 5276 raft_consensus.cc:2258] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.676265 5274 raft_consensus.cc:2258] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.676978 5276 ts_tablet_manager.cc:1905] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.677474 5274 ts_tablet_manager.cc:1905] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.679850 5276 ts_tablet_manager.cc:1918] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:37.679916 5276 log.cc:1196] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/c87b4f88c6e542d0ab328917709cdc91
I20260928 02:46:37.680078 5274 ts_tablet_manager.cc:1918] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:37.680140 5274 log.cc:1196] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/c87b4f88c6e542d0ab328917709cdc91
I20260928 02:46:37.680171 5276 ts_tablet_manager.cc:1939] T c87b4f88c6e542d0ab328917709cdc91 P e2f8468eb794442c92d916a922f8f584: Deleting consensus metadata
I20260928 02:46:37.680378 5274 ts_tablet_manager.cc:1939] T c87b4f88c6e542d0ab328917709cdc91 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting consensus metadata
I20260928 02:46:37.680943 4724 catalog_manager.cc:4841] TS e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331): tablet c87b4f88c6e542d0ab328917709cdc91 (table test-table [id=bd40602252fa4f6f90471d2131d387cc]) successfully deleted
I20260928 02:46:37.680991 4725 catalog_manager.cc:4841] TS 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131): tablet c87b4f88c6e542d0ab328917709cdc91 (table test-table [id=bd40602252fa4f6f90471d2131d387cc]) successfully deleted
I20260928 02:46:37.681546 5157 tablet_service.cc:1514] Processing DeleteTablet for tablet c87b4f88c6e542d0ab328917709cdc91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:50850
I20260928 02:46:37.681743 5275 tablet_replica.cc:330] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: stopping tablet replica
I20260928 02:46:37.681877 5275 raft_consensus.cc:2229] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:37.682001 5275 raft_consensus.cc:2258] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.682796 5275 ts_tablet_manager.cc:1905] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.685791 5275 ts_tablet_manager.cc:1918] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:37.685866 5275 log.cc:1196] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/c87b4f88c6e542d0ab328917709cdc91
I20260928 02:46:37.686120 5275 ts_tablet_manager.cc:1939] T c87b4f88c6e542d0ab328917709cdc91 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting consensus metadata
I20260928 02:46:37.686770 4726 catalog_manager.cc:4841] TS 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603): tablet c87b4f88c6e542d0ab328917709cdc91 (table test-table [id=bd40602252fa4f6f90471d2131d387cc]) successfully deleted
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:46:37.713128 4789 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:37.705111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.28.248.190:37681)
0928 02:46:37.705241 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.705246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.705508 (+ 262us) server_negotiation.cc:407] Connection header received
0928 02:46:37.705665 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.705669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.705729 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.705827 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:37.706706 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.707513 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.708811 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.709148 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.709260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.713000 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47892: BlockingRecv error: recv got EOF from 127.0.0.1:47892 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 02:46:37 dist-test-slave-r238 krb5kdc[4695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM
I20260928 02:46:37.726903 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.716988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.28.248.190:37681)
0928 02:46:37.717094 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.717099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.717275 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:46:37.717375 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.717378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.717423 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.717521 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:37.718213 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.719031 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.720148 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.720488 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.720587 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.723381 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:37.723385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:37.723388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:37.723414 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:37.725191 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.725665 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.725668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.725671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.725756 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:37.726038 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:37.726042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:37.726043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:37.726246 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:37.726371 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.726625 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.726738 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:46:37.730314 4738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:37.730787 4738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:37.735610 4866 tablet_service.cc:1467] Processing CreateTablet for tablet 6de679c12d2b4efb9d600e16219085a9 (DEFAULT_TABLE table=test-table [id=77f03a814f80467f940bf4dd15b2764d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.735912 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de679c12d2b4efb9d600e16219085a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.736909 5157 tablet_service.cc:1467] Processing CreateTablet for tablet 6de679c12d2b4efb9d600e16219085a9 (DEFAULT_TABLE table=test-table [id=77f03a814f80467f940bf4dd15b2764d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.737176 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de679c12d2b4efb9d600e16219085a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.739518 5252 tablet_bootstrap.cc:492] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Bootstrap starting.
I20260928 02:46:37.740149 5017 tablet_service.cc:1467] Processing CreateTablet for tablet 6de679c12d2b4efb9d600e16219085a9 (DEFAULT_TABLE table=test-table [id=77f03a814f80467f940bf4dd15b2764d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.740389 5252 tablet_bootstrap.cc:654] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.740397 5017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de679c12d2b4efb9d600e16219085a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.742615 5251 tablet_bootstrap.cc:492] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Bootstrap starting.
I20260928 02:46:37.743227 5255 tablet_bootstrap.cc:492] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Bootstrap starting.
I20260928 02:46:37.743573 5251 tablet_bootstrap.cc:654] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.743966 5255 tablet_bootstrap.cc:654] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.744099 5252 tablet_bootstrap.cc:492] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: No bootstrap required, opened a new log
I20260928 02:46:37.744201 5252 ts_tablet_manager.cc:1397] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 02:46:37.744740 5252 raft_consensus.cc:348] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.744853 5252 raft_consensus.cc:374] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.744885 5252 raft_consensus.cc:729] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19496ebbb86243eab7ba76d2d8faa7ce, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.745054 5255 tablet_bootstrap.cc:492] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: No bootstrap required, opened a new log
I20260928 02:46:37.745021 5252 consensus_queue.cc:260] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.745124 5255 ts_tablet_manager.cc:1397] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:37.745532 5255 raft_consensus.cc:348] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.745635 5255 raft_consensus.cc:374] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.745664 5255 raft_consensus.cc:729] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.745662 5251 tablet_bootstrap.cc:492] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: No bootstrap required, opened a new log
I20260928 02:46:37.745731 5251 ts_tablet_manager.cc:1397] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:37.745779 5255 consensus_queue.cc:260] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.745966 5252 ts_tablet_manager.cc:1428] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:46:37.746026 5255 ts_tablet_manager.cc:1428] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.746165 5251 raft_consensus.cc:348] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.746277 5251 raft_consensus.cc:374] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.746307 5251 raft_consensus.cc:729] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.746421 5251 consensus_queue.cc:260] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.746665 5251 ts_tablet_manager.cc:1428] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.916435 5305 raft_consensus.cc:482] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:37.916576 5305 raft_consensus.cc:504] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.917058 5305 leader_election.cc:288] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.917546 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de679c12d2b4efb9d600e16219085a9" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" is_pre_election: true
I20260928 02:46:37.917546 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de679c12d2b4efb9d600e16219085a9" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f8468eb794442c92d916a922f8f584" is_pre_election: true
I20260928 02:46:37.917702 5170 raft_consensus.cc:2454] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 0.
I20260928 02:46:37.917702 4879 raft_consensus.cc:2454] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 0.
I20260928 02:46:37.918017 4968 leader_election.cc:302] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d910e3d7604bbca93ea792c3a9cc1f, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.918229 5305 raft_consensus.cc:2781] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:37.918295 5305 raft_consensus.cc:482] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:37.918324 5305 raft_consensus.cc:3037] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.919286 5305 raft_consensus.cc:504] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.919596 5305 leader_election.cc:288] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 election: Requested vote from peers e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331), 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603)
I20260928 02:46:37.919883 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de679c12d2b4efb9d600e16219085a9" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f8468eb794442c92d916a922f8f584"
I20260928 02:46:37.920013 4879 raft_consensus.cc:3037] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.919993 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de679c12d2b4efb9d600e16219085a9" candidate_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19496ebbb86243eab7ba76d2d8faa7ce"
I20260928 02:46:37.920140 5170 raft_consensus.cc:3037] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:37.921068 4879 raft_consensus.cc:2454] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 1.
I20260928 02:46:37.921103 5170 raft_consensus.cc:2454] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26d910e3d7604bbca93ea792c3a9cc1f in term 1.
I20260928 02:46:37.921371 4968 leader_election.cc:302] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d910e3d7604bbca93ea792c3a9cc1f, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:37.921546 5305 raft_consensus.cc:2781] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:37.921660 5305 raft_consensus.cc:686] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 LEADER]: Becoming Leader. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Running, Role: LEADER
I20260928 02:46:37.921798 5305 consensus_queue.cc:237] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.923472 4739 catalog_manager.cc:5495] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130). New cstate: current_term: 1 leader_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:37.937405 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:37.934236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.28.248.130:42131)
0928 02:46:37.934369 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:37.934375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:37.934390 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:37.934560 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:37.934563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:37.934613 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:37.934689 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:37.934934 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.935470 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.936301 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:37.936522 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:37.936645 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:37.936897 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:37.936963 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:37.937051 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:37.937205 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:46:37.937252 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:37.939173 4879 raft_consensus.cc:1261] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Refusing update from remote peer 26d910e3d7604bbca93ea792c3a9cc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.939179 5170 raft_consensus.cc:1261] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Refusing update from remote peer 26d910e3d7604bbca93ea792c3a9cc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:37.939564 5258 consensus_queue.cc:1035] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.939719 5305 consensus_queue.cc:1035] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:37.947024 4738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:37.947180 4738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:37.949242 4738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Sending DeleteTablet for 3 replicas of tablet 6de679c12d2b4efb9d600e16219085a9
I20260928 02:46:37.949692 5017 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de679c12d2b4efb9d600e16219085a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51498
I20260928 02:46:37.949801 5016 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de679c12d2b4efb9d600e16219085a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51498
I20260928 02:46:37.949854 5274 tablet_replica.cc:330] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: stopping tablet replica
I20260928 02:46:37.949973 5274 raft_consensus.cc:2229] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:37.949978 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de679c12d2b4efb9d600e16219085a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55390
I20260928 02:46:37.950167 5276 tablet_replica.cc:330] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: stopping tablet replica
I20260928 02:46:37.950160 5274 raft_consensus.cc:2258] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.950294 5276 raft_consensus.cc:2229] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:37.950299 4738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:46:37.950412 5276 raft_consensus.cc:2258] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.950588 5157 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de679c12d2b4efb9d600e16219085a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:37 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:50850
W20260928 02:46:37.950742 4738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:37.950764 5275 tablet_replica.cc:330] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: stopping tablet replica
I20260928 02:46:37.950878 5275 raft_consensus.cc:2229] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:37.951014 5275 raft_consensus.cc:2258] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:37.951244 5276 ts_tablet_manager.cc:1905] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.951670 5275 ts_tablet_manager.cc:1905] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.953369 5276 ts_tablet_manager.cc:1918] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:37.953430 5276 log.cc:1196] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/6de679c12d2b4efb9d600e16219085a9
I20260928 02:46:37.953675 5276 ts_tablet_manager.cc:1939] T 6de679c12d2b4efb9d600e16219085a9 P e2f8468eb794442c92d916a922f8f584: Deleting consensus metadata
I20260928 02:46:37.953806 5275 ts_tablet_manager.cc:1918] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:37.953904 5275 log.cc:1196] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/6de679c12d2b4efb9d600e16219085a9
I20260928 02:46:37.954171 5275 ts_tablet_manager.cc:1939] T 6de679c12d2b4efb9d600e16219085a9 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting consensus metadata
I20260928 02:46:37.954280 4724 catalog_manager.cc:4841] TS e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331): tablet 6de679c12d2b4efb9d600e16219085a9 (table test-table [id=77f03a814f80467f940bf4dd15b2764d]) successfully deleted
I20260928 02:46:37.954552 5274 ts_tablet_manager.cc:1905] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:37.954762 4726 catalog_manager.cc:4841] TS 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603): tablet 6de679c12d2b4efb9d600e16219085a9 (table test-table [id=77f03a814f80467f940bf4dd15b2764d]) successfully deleted
I20260928 02:46:37.956666 5274 ts_tablet_manager.cc:1918] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:37.956746 5274 log.cc:1196] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/6de679c12d2b4efb9d600e16219085a9
I20260928 02:46:37.956996 5274 ts_tablet_manager.cc:1939] T 6de679c12d2b4efb9d600e16219085a9 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting consensus metadata
I20260928 02:46:37.957629 4725 catalog_manager.cc:4841] TS 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131): tablet 6de679c12d2b4efb9d600e16219085a9 (table test-table [id=77f03a814f80467f940bf4dd15b2764d]) successfully deleted
W20260928 02:46:37.958060 4725 catalog_manager.cc:4808] TS 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131): delete failed for tablet 6de679c12d2b4efb9d600e16219085a9 because the tablet was not found. No further retry: Not found: Tablet not found: 6de679c12d2b4efb9d600e16219085a9
I20260928 02:46:37.958419 4866 tablet_service.cc:1467] Processing CreateTablet for tablet ea2b1f9e446a4203b4837bf6abbd4053 (DEFAULT_TABLE table=test-table [id=b09d5d22fe624cac9ca1a3e015e4a3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.958576 5016 tablet_service.cc:1467] Processing CreateTablet for tablet ea2b1f9e446a4203b4837bf6abbd4053 (DEFAULT_TABLE table=test-table [id=b09d5d22fe624cac9ca1a3e015e4a3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.958679 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2b1f9e446a4203b4837bf6abbd4053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.958808 5016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2b1f9e446a4203b4837bf6abbd4053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.960868 5157 tablet_service.cc:1467] Processing CreateTablet for tablet ea2b1f9e446a4203b4837bf6abbd4053 (DEFAULT_TABLE table=test-table [id=b09d5d22fe624cac9ca1a3e015e4a3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:37.960928 5255 tablet_bootstrap.cc:492] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584: Bootstrap starting.
I20260928 02:46:37.961117 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2b1f9e446a4203b4837bf6abbd4053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:37.961664 5255 tablet_bootstrap.cc:654] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.962242 5251 tablet_bootstrap.cc:492] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Bootstrap starting.
I20260928 02:46:37.962787 5255 tablet_bootstrap.cc:492] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584: No bootstrap required, opened a new log
I20260928 02:46:37.962872 5255 ts_tablet_manager.cc:1397] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:37.963054 5251 tablet_bootstrap.cc:654] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.963125 5252 tablet_bootstrap.cc:492] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Bootstrap starting.
I20260928 02:46:37.963299 5255 raft_consensus.cc:348] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.963441 5255 raft_consensus.cc:374] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.963495 5255 raft_consensus.cc:729] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f8468eb794442c92d916a922f8f584, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.963624 5255 consensus_queue.cc:260] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.963912 5255 ts_tablet_manager.cc:1428] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.964040 5251 tablet_bootstrap.cc:492] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: No bootstrap required, opened a new log
I20260928 02:46:37.964135 5251 ts_tablet_manager.cc:1397] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:37.964464 5252 tablet_bootstrap.cc:654] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:37.964538 5251 raft_consensus.cc:348] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.964699 5251 raft_consensus.cc:374] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.964738 5251 raft_consensus.cc:729] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d910e3d7604bbca93ea792c3a9cc1f, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.964855 5251 consensus_queue.cc:260] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.965152 5251 ts_tablet_manager.cc:1428] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:37.966181 5252 tablet_bootstrap.cc:492] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: No bootstrap required, opened a new log
I20260928 02:46:37.966265 5252 ts_tablet_manager.cc:1397] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:46:37.966678 5252 raft_consensus.cc:348] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.966763 5252 raft_consensus.cc:374] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:37.966785 5252 raft_consensus.cc:729] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19496ebbb86243eab7ba76d2d8faa7ce, State: Initialized, Role: FOLLOWER
I20260928 02:46:37.966900 5252 consensus_queue.cc:260] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:37.967180 5252 ts_tablet_manager.cc:1428] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:38.091071 5257 raft_consensus.cc:482] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:38.091236 5257 raft_consensus.cc:504] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:38.092352 5257 leader_election.cc:288] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331), 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131)
I20260928 02:46:38.096319 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.092736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:42811 (local address 127.28.248.129:42331)
0928 02:46:38.092880 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:38.092887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:38.093084 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:46:38.093190 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:38.093194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:38.093242 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:38.093322 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:38.093611 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.094172 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.095356 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.095745 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.095875 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.095931 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:38.096119 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:46:38.096187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:46:38.096321 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.092817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:42167 (local address 127.28.248.130:42131)
0928 02:46:38.092963 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:38.092968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:38.093212 (+ 244us) server_negotiation.cc:407] Connection header received
0928 02:46:38.093341 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:38.093345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:38.093389 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:38.093466 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:38.093723 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.094200 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.095354 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.095744 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.095878 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.095931 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:38.096150 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:46:38.096215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:46:38.096333 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.092627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:42131 (local address 127.28.248.131:42167)
0928 02:46:38.093101 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:38.093116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:38.093218 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:38.093486 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:38.093488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:38.093514 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:38.093618 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.093624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.094333 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.094335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.095232 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.095241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.095872 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.095875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.096013 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.096030 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:46:38.096122 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:46:38.096187 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":162,"thread_start_us":44,"threads_started":1}
I20260928 02:46:38.096339 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.092627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.28.248.131:42811)
0928 02:46:38.092987 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:38.093009 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:38.093102 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:38.093342 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:38.093345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:38.093362 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:38.093493 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.093501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.094302 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.094306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.095234 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.095243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.095872 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.095875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.096001 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.096019 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 02:46:38.096078 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:46:38.096167 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":69,"thread_start_us":80,"threads_started":1}
I20260928 02:46:38.096801 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b1f9e446a4203b4837bf6abbd4053" candidate_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" is_pre_election: true
I20260928 02:46:38.096954 5037 raft_consensus.cc:2454] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19496ebbb86243eab7ba76d2d8faa7ce in term 0.
I20260928 02:46:38.097218 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b1f9e446a4203b4837bf6abbd4053" candidate_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f8468eb794442c92d916a922f8f584" is_pre_election: true
I20260928 02:46:38.097312 5109 leader_election.cc:302] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19496ebbb86243eab7ba76d2d8faa7ce, 26d910e3d7604bbca93ea792c3a9cc1f; no voters:
I20260928 02:46:38.097368 4879 raft_consensus.cc:2454] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19496ebbb86243eab7ba76d2d8faa7ce in term 0.
I20260928 02:46:38.097560 5257 raft_consensus.cc:2781] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:38.097621 5257 raft_consensus.cc:482] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:38.097671 5257 raft_consensus.cc:3037] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:38.098618 5257 raft_consensus.cc:504] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:38.098977 5257 leader_election.cc:288] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [CANDIDATE]: Term 1 election: Requested vote from peers e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331), 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131)
I20260928 02:46:38.099314 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b1f9e446a4203b4837bf6abbd4053" candidate_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f8468eb794442c92d916a922f8f584"
I20260928 02:46:38.099445 4879 raft_consensus.cc:3037] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:38.099421 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b1f9e446a4203b4837bf6abbd4053" candidate_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d910e3d7604bbca93ea792c3a9cc1f"
I20260928 02:46:38.099566 5037 raft_consensus.cc:3037] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:38.100387 4879 raft_consensus.cc:2454] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19496ebbb86243eab7ba76d2d8faa7ce in term 1.
I20260928 02:46:38.100481 5037 raft_consensus.cc:2454] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19496ebbb86243eab7ba76d2d8faa7ce in term 1.
I20260928 02:46:38.100728 5108 leader_election.cc:302] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19496ebbb86243eab7ba76d2d8faa7ce, e2f8468eb794442c92d916a922f8f584; no voters:
I20260928 02:46:38.100888 5257 raft_consensus.cc:2781] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:38.101087 5257 raft_consensus.cc:686] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 LEADER]: Becoming Leader. State: Replica: 19496ebbb86243eab7ba76d2d8faa7ce, State: Running, Role: LEADER
I20260928 02:46:38.101358 5257 consensus_queue.cc:237] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } }
I20260928 02:46:38.103487 4740 catalog_manager.cc:5495] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131). New cstate: current_term: 1 leader_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19496ebbb86243eab7ba76d2d8faa7ce" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41603 } health_report { overall_health: HEALTHY } } }
I20260928 02:46:38.120234 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.116517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.28.248.131:41603)
0928 02:46:38.116777 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:38.116787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:38.116805 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:46:38.116880 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:38.116883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:38.116941 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:38.117089 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:38.117318 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.117854 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.118679 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.118932 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.119057 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.119315 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:38.119815 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:38.119942 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:38.120012 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:46:38.120063 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260928 02:46:38.129202 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.126422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:39311 (local address 127.28.248.129:42331)
0928 02:46:38.126542 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:38.126545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:38.126650 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:46:38.126743 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:38.126745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:38.126784 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:38.126837 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:38.127039 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.127508 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.128308 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.128512 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.128582 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.128877 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:38.128936 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:38.128990 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:38.129067 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:46:38.129105 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:46:38.129236 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.126311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.28.248.131:39311)
0928 02:46:38.126578 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:38.126590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:38.126668 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:38.126859 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:38.126861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:38.126867 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:38.126953 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.126956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.127594 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.127597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.128215 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.128223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.128599 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.128601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.128726 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.128765 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:38.129008 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:38.129012 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:38.129060 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:46:38.129120 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260928 02:46:38.134649 5037 raft_consensus.cc:1261] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Refusing update from remote peer 19496ebbb86243eab7ba76d2d8faa7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:38.134795 4879 raft_consensus.cc:1261] T ea2b1f9e446a4203b4837bf6abbd4053 P e2f8468eb794442c92d916a922f8f584 [term 1 FOLLOWER]: Refusing update from remote peer 19496ebbb86243eab7ba76d2d8faa7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:38.135098 5257 consensus_queue.cc:1035] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d910e3d7604bbca93ea792c3a9cc1f" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:38.135264 5334 consensus_queue.cc:1035] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f8468eb794442c92d916a922f8f584" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:38.152835 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.149139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:44395 (local address 127.28.248.131:41603)
0928 02:46:38.149250 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:38.149256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:38.149404 (+ 148us) server_negotiation.cc:407] Connection header received
0928 02:46:38.149544 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:38.149546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:38.149591 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:38.149670 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:38.149926 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.150575 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.151726 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.152039 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.152144 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.152377 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:38.152460 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:38.152539 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:38.152676 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:46:38.152736 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:46:38.152848 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.149098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41603 (local address 127.28.248.129:44395)
0928 02:46:38.149313 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:38.149327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:38.149447 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:38.149689 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:38.149692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:38.149700 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:46:38.149803 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.149809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.150681 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.150686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.151609 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:38.151618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:38.152155 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:38.152158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:38.152242 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:38.152277 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:46:38.152543 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:46:38.152551 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:38.152646 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:46:38.152732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260928 02:46:38.172781 4738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:38.172919 4738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:38.174940 4738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e0c366c2f6c40fa8d8087fad87fa655: Sending DeleteTablet for 3 replicas of tablet ea2b1f9e446a4203b4837bf6abbd4053
I20260928 02:46:38.175581 5157 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2b1f9e446a4203b4837bf6abbd4053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:38 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:50850
I20260928 02:46:38.175695 5016 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2b1f9e446a4203b4837bf6abbd4053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:38 UTC) from {username='oryx', principal='oryx/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51498
I20260928 02:46:38.175796 5275 tablet_replica.cc:330] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: stopping tablet replica
I20260928 02:46:38.175865 5274 tablet_replica.cc:330] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: stopping tablet replica
I20260928 02:46:38.175911 5275 raft_consensus.cc:2229] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:38.175956 5274 raft_consensus.cc:2229] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:38.176047 5274 raft_consensus.cc:2258] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:38.176193 5275 raft_consensus.cc:2258] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:38.176246 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 4812
I20260928 02:46:38.177918 5274 ts_tablet_manager.cc:1905] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:38.178483 5275 ts_tablet_manager.cc:1905] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:38.180486 5274 ts_tablet_manager.cc:1918] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:38.180568 5274 log.cc:1196] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/ea2b1f9e446a4203b4837bf6abbd4053
I20260928 02:46:38.180953 5275 ts_tablet_manager.cc:1918] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:46:38.181038 5275 log.cc:1196] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/ea2b1f9e446a4203b4837bf6abbd4053
I20260928 02:46:38.181264 5274 ts_tablet_manager.cc:1939] T ea2b1f9e446a4203b4837bf6abbd4053 P 26d910e3d7604bbca93ea792c3a9cc1f: Deleting consensus metadata
I20260928 02:46:38.181283 5275 ts_tablet_manager.cc:1939] T ea2b1f9e446a4203b4837bf6abbd4053 P 19496ebbb86243eab7ba76d2d8faa7ce: Deleting consensus metadata
I20260928 02:46:38.181964 4725 catalog_manager.cc:4841] TS 26d910e3d7604bbca93ea792c3a9cc1f (127.28.248.130:42131): tablet ea2b1f9e446a4203b4837bf6abbd4053 (table test-table [id=b09d5d22fe624cac9ca1a3e015e4a3e3]) successfully deleted
I20260928 02:46:38.182060 4726 catalog_manager.cc:4841] TS 19496ebbb86243eab7ba76d2d8faa7ce (127.28.248.131:41603): tablet ea2b1f9e446a4203b4837bf6abbd4053 (table test-table [id=b09d5d22fe624cac9ca1a3e015e4a3e3]) successfully deleted
W20260928 02:46:38.187885 4724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:42331 (error 108)
I20260928 02:46:38.188203 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 4952
I20260928 02:46:38.190745 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:38.190285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:42331 (local address 127.0.0.1:55418)
0928 02:46:38.190535 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:38.190619 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:42331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
W20260928 02:46:38.190971 4724 catalog_manager.cc:4584] TS e2f8468eb794442c92d916a922f8f584 (127.28.248.129:42331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea2b1f9e446a4203b4837bf6abbd4053: Network error: Client connection negotiation failed: client connection to 127.28.248.129:42331: connect: Connection refused (error 111)
I20260928 02:46:38.195617 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5092
I20260928 02:46:38.200721 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 4708
2026-09-28T02:46:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3267 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:38 dist-test-slave-r238 krb5kdc[5350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:38 dist-test-slave-r238 krb5kdc[5350](info): set up 2 sockets
Sep 28 02:46:38 dist-test-slave-r238 krb5kdc[5350](info): commencing operation
krb5kdc: starting...
W20260928 02:46:39.245287 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:39 dist-test-slave-r238 krb5kdc[5350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:39.364141 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:44561
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:39.474128 5363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:39.474453 5363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:39.474540 5363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:39.478502 5363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:39.478595 5363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:39.478633 5363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:39.478691 5363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:39.478729 5363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:39.483915 5363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:44561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:39.484954 5363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:39.485961 5363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:39.494388 5369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:39.494355 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:39.494612 5363 server_base.cc:1034] running on GCE node
W20260928 02:46:39.494376 5368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:39.495225 5363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:39.496467 5363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:39.497702 5363 hybrid_clock.cc:648] HybridClock initialized: now 1790563599497690 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:46:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1300 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:39 dist-test-slave-r238 krb5kdc[5376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:39 dist-test-slave-r238 krb5kdc[5376](info): set up 2 sockets
Sep 28 02:46:39 dist-test-slave-r238 krb5kdc[5376](info): commencing operation
krb5kdc: starting...
W20260928 02:46:40.569831 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:46:40 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:46:40.695222 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39227
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:40.809679 5389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:40.809971 5389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:40.810014 5389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:40.814083 5389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:40.814149 5389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:40.814170 5389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:40.814186 5389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:40.814203 5389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:40.819401 5389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:40.820407 5389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:40.821406 5389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:40.829702 5394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:40.829701 5395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:40.829722 5397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:40.830016 5389 server_base.cc:1034] running on GCE node
I20260928 02:46:40.830610 5389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:40.831652 5389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:40.832839 5389 hybrid_clock.cc:648] HybridClock initialized: now 1790563600832804 us; error 47 us; skew 500 ppm
Sep 28 02:46:40 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563600, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:40.840917 5389 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:40.841928 5389 webserver.cc:460] Webserver started at http://127.28.248.190:42093/ using document root <none> and password file <none>
I20260928 02:46:40.842549 5389 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:40.842643 5389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:40.842882 5389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:40.845142 5389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "65a0b770a1164ae5bacfdf0388396ceb"
format_stamp: "Formatted at 2026-09-28 02:46:40 on dist-test-slave-r238"
server_key: "14837d3198b9c21579a933be4a00fe89"
server_key_iv: "c4ab05e1f655dca6a51751fc902e4250"
server_key_version: "encryptionkey@0"
I20260928 02:46:40.845763 5389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "65a0b770a1164ae5bacfdf0388396ceb"
format_stamp: "Formatted at 2026-09-28 02:46:40 on dist-test-slave-r238"
server_key: "14837d3198b9c21579a933be4a00fe89"
server_key_iv: "c4ab05e1f655dca6a51751fc902e4250"
server_key_version: "encryptionkey@0"
I20260928 02:46:40.849582 5389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:46:40.852042 5404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:40.853477 5389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:40.853636 5389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "65a0b770a1164ae5bacfdf0388396ceb"
format_stamp: "Formatted at 2026-09-28 02:46:40 on dist-test-slave-r238"
server_key: "14837d3198b9c21579a933be4a00fe89"
server_key_iv: "c4ab05e1f655dca6a51751fc902e4250"
server_key_version: "encryptionkey@0"
I20260928 02:46:40.853755 5389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:40.868934 5389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:40.872546 5389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:40.872809 5389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:40.880398 5389 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:39227
I20260928 02:46:40.880412 5456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:39227 every 8 connection(s)
I20260928 02:46:40.881479 5389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:40.881717 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5389
I20260928 02:46:40.881850 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:40.882246 29666 external_mini_cluster.cc:1428] Setting key 3ea9571bb293e83f53831994602ad4a3
I20260928 02:46:40.884919 5457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:40.890975 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Bootstrap starting.
Sep 28 02:46:40 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:40.893851 5457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:40.894791 5457 log.cc:824] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:40.897197 5457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: No bootstrap required, opened a new log
I20260928 02:46:40.897541 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:40.883963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.28.248.190:39227)
0928 02:46:40.884364 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:40.884382 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:40.884429 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:46:40.885228 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:40.885248 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:40.885707 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:40.886070 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:40.886861 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:40.887846 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:40.888798 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:40.889519 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:40.889711 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:40.892134 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:40.892144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:40.892160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:40.892195 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:40.894310 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:40.894735 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:40.894741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:40.894747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:40.894909 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:40.895240 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:40.895243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:40.895245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:40.895797 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:40.895993 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:40.896225 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 02:46:40.896518 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":153,"threads_started":1}
I20260928 02:46:40.900830 5457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:40.901134 5457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:40.901218 5457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a0b770a1164ae5bacfdf0388396ceb, State: Initialized, Role: FOLLOWER
I20260928 02:46:40.901721 5457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:40.901881 5457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:40.901957 5457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:40.902060 5457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:40.903354 5457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:40.903735 5457 leader_election.cc:302] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a0b770a1164ae5bacfdf0388396ceb; no voters:
I20260928 02:46:40.904114 5457 leader_election.cc:288] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:40.904345 5462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:40.904606 5462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 LEADER]: Becoming Leader. State: Replica: 65a0b770a1164ae5bacfdf0388396ceb, State: Running, Role: LEADER
I20260928 02:46:40.904927 5462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:40.905495 5457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:40.906574 5463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a0b770a1164ae5bacfdf0388396ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } } }
I20260928 02:46:40.906700 5463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:40.907143 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a0b770a1164ae5bacfdf0388396ceb. Latest consensus state: current_term: 1 leader_uuid: "65a0b770a1164ae5bacfdf0388396ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } } }
I20260928 02:46:40.907223 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: This master's current role is: LEADER
W20260928 02:46:40.910430 5477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:40.910557 5477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:40.910637 5471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:40.911544 5471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:40.916941 5471 catalog_manager.cc:1347] Generated new cluster ID: 5406fcf369e042f0a3a24e96c925798e
I20260928 02:46:40.917018 5471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:40.926577 5471 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:40.927470 5471 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:40.936987 5471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Generated new TSK 0
I20260928 02:46:40.937706 5471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:46:40.971848 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39227
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:41.114135 5483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:41.114385 5483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:41.114429 5483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:41.118307 5483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:41.118361 5483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:41.118417 5483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:41.123551 5483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:41.124543 5483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:41.125604 5483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:41.135150 5488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:41.135142 5489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:41.135162 5491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:41.135210 5483 server_base.cc:1034] running on GCE node
I20260928 02:46:41.135764 5483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:41.136399 5483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:41.137570 5483 hybrid_clock.cc:648] HybridClock initialized: now 1790563601137525 us; error 58 us; skew 500 ppm
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:41.140216 5483 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:46:41.141119 5483 webserver.cc:460] Webserver started at http://127.28.248.129:41365/ using document root <none> and password file <none>
I20260928 02:46:41.141680 5483 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:41.141767 5483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:41.141990 5483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:41.144042 5483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "248983fa9a544cfd946af571c35bc946"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "44c5bd39dd67011c44864bd24de6122e"
server_key_iv: "cb41597a1508655edc797f8b270b0b7c"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.144670 5483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "248983fa9a544cfd946af571c35bc946"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "44c5bd39dd67011c44864bd24de6122e"
server_key_iv: "cb41597a1508655edc797f8b270b0b7c"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.148398 5483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:46:41.150892 5498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.152105 5483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:41.152235 5483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "248983fa9a544cfd946af571c35bc946"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "44c5bd39dd67011c44864bd24de6122e"
server_key_iv: "cb41597a1508655edc797f8b270b0b7c"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.152343 5483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:41.163457 5483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:41.166389 5483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:41.166604 5483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:41.167148 5483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:41.167919 5483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:41.167987 5483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.168053 5483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:41.168099 5483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.177224 5483 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:45115
I20260928 02:46:41.177248 5611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:45115 every 8 connection(s)
I20260928 02:46:41.178078 5483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:41.179246 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5483
I20260928 02:46:41.179382 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:41.179706 29666 external_mini_cluster.cc:1428] Setting key 6eef9713f74d2b366eac61f867cc3804
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:46:41.194141 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:41.179987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:33403 (local address 127.28.248.190:39227)
0928 02:46:41.180135 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:41.180138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:41.181170 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 02:46:41.183206 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:41.183210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:41.183269 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:41.183366 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:41.184577 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.185574 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.186788 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.187262 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.187394 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:41.189997 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:41.190000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:41.190002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:41.190031 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:41.191793 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:41.192341 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:41.192348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:41.192350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:41.192446 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:41.192939 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:41.192943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:41.192945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:41.193172 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:41.193285 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:41.193812 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 02:46:41.193946 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:41.194975 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:41.180412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:33403)
0928 02:46:41.181005 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:41.181062 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:41.182939 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:41.183502 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:41.183509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:41.183845 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:41.184419 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:41.184434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.185724 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.185729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:41.186652 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:41.186661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.187437 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.187440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:41.187606 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:41.188271 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:41.188288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:41.189825 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:41.191912 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:41.191917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:41.191932 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:41.192207 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:41.192587 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:41.192593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:41.192598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:41.192832 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:41.193282 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:41.193290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:41.193566 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 02:46:41.193758 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:46:41.194104 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":204,"threads_started":1}
I20260928 02:46:41.196339 5612 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39227
I20260928 02:46:41.196662 5612 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:41.197181 5612 heartbeater.cc:511] Master 127.28.248.190:39227 requested a full tablet report, sending...
I20260928 02:46:41.198982 5421 ts_manager.cc:194] Registered new tserver with Master: 248983fa9a544cfd946af571c35bc946 (127.28.248.129:45115)
I20260928 02:46:41.200727 5421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:33403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:46:41.211443 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39227
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:41.323231 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:41.323526 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:41.323614 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:41.327608 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:41.327699 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:41.327795 5617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:41.332909 5617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:41.333904 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:41.334884 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:41.344359 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:41.344403 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:41.344425 5617 server_base.cc:1034] running on GCE node
W20260928 02:46:41.344416 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:41.345072 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:41.345688 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:41.346886 5617 hybrid_clock.cc:648] HybridClock initialized: now 1790563601346858 us; error 31 us; skew 500 ppm
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:41.349630 5617 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:46:41.350389 5617 webserver.cc:460] Webserver started at http://127.28.248.130:34017/ using document root <none> and password file <none>
I20260928 02:46:41.350899 5617 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:41.350958 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:41.351114 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:41.353206 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "f9a255af82444751a0c0b10a9188b9e4"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "9ab0dfe1e1f2658736bb28a814a793e1"
server_key_iv: "787c25b377651725b48ad04b5835544d"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.353699 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "f9a255af82444751a0c0b10a9188b9e4"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "9ab0dfe1e1f2658736bb28a814a793e1"
server_key_iv: "787c25b377651725b48ad04b5835544d"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.357381 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:46:41.359695 5632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.360915 5617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:41.361037 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "f9a255af82444751a0c0b10a9188b9e4"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "9ab0dfe1e1f2658736bb28a814a793e1"
server_key_iv: "787c25b377651725b48ad04b5835544d"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.361131 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:41.387984 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:41.391276 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:41.391485 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:41.392052 5617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:41.392860 5617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:41.392905 5617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.392971 5617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:41.393007 5617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.401911 5617 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:34483
I20260928 02:46:41.401935 5745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:34483 every 8 connection(s)
I20260928 02:46:41.402741 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:41.407826 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5617
I20260928 02:46:41.407948 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:41.408279 29666 external_mini_cluster.cc:1428] Setting key b09af5cbcbd84fad1c9102823e8db9cb
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:41.417096 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:41.404459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:40501 (local address 127.28.248.190:39227)
0928 02:46:41.404653 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:41.404657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:41.405484 (+ 827us) server_negotiation.cc:407] Connection header received
0928 02:46:41.406997 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:41.406999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:41.407059 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:41.407146 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:41.408416 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.409123 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.410418 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.410839 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.410965 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:41.413611 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:41.413615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:41.413617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:41.413649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:41.415088 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:41.415596 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:41.415601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:41.415603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:41.415683 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:41.416014 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:41.416019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:41.416024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:41.416196 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:41.416299 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:41.416824 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 02:46:41.416938 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:46:41.417883 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:41.404880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:40501)
0928 02:46:41.405340 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:41.405391 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:41.406811 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:41.407314 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:41.407321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:41.407709 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:41.408242 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:41.408257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.409244 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.409248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:41.410293 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:41.410303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.410997 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.411000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:41.411132 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:41.411763 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:41.411787 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:41.413452 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:41.415220 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:41.415226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:41.415239 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:41.415493 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:41.415783 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:41.415786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:41.415788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:41.415920 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:41.416309 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:41.416317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:41.416548 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 02:46:41.416783 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:46:41.417093 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":150,"threads_started":1}
I20260928 02:46:41.419062 5746 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39227
I20260928 02:46:41.419317 5746 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:46:41.419826 5746 heartbeater.cc:511] Master 127.28.248.190:39227 requested a full tablet report, sending...
I20260928 02:46:41.420946 5421 ts_manager.cc:194] Registered new tserver with Master: f9a255af82444751a0c0b10a9188b9e4 (127.28.248.130:34483)
I20260928 02:46:41.421663 5421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:40501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:46:41.433454 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39227
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:41.555166 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:41.555418 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:41.555518 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:41.559739 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:41.559793 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:41.559904 5751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:41.565438 5751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:41.566465 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:41.567466 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:41.577272 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:41.577399 5751 server_base.cc:1034] running on GCE node
W20260928 02:46:41.577260 5757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:41.577262 5756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:41.577960 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:41.578660 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:41.579850 5751 hybrid_clock.cc:648] HybridClock initialized: now 1790563601579835 us; error 25 us; skew 500 ppm
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:41.582757 5751 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:46:41.583679 5751 webserver.cc:460] Webserver started at http://127.28.248.131:40085/ using document root <none> and password file <none>
I20260928 02:46:41.584303 5751 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:41.584419 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:41.584725 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:41.586891 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "03fdeb2f1d034bfab6e11896e97f694a"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "20927c48c9a7aa19687035e97ec0ed3d"
server_key_iv: "f34d61668ba6190140460a521d23ffd0"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.587477 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "03fdeb2f1d034bfab6e11896e97f694a"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "20927c48c9a7aa19687035e97ec0ed3d"
server_key_iv: "f34d61668ba6190140460a521d23ffd0"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.591382 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:46:41.593927 5766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.595077 5751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:41.595219 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "03fdeb2f1d034bfab6e11896e97f694a"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "20927c48c9a7aa19687035e97ec0ed3d"
server_key_iv: "f34d61668ba6190140460a521d23ffd0"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.595342 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:41.608119 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:41.611003 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:41.611197 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:41.611758 5751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:41.612545 5751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:41.612589 5751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.612689 5751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:41.612726 5751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.621096 5751 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:32911
I20260928 02:46:41.621181 5879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:32911 every 8 connection(s)
I20260928 02:46:41.621979 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:41.629154 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5751
I20260928 02:46:41.629311 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:41.629608 29666 external_mini_cluster.cc:1428] Setting key 0ab85662e38d8033425a1fc354eac717
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:46:41.636116 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:41.623824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54847 (local address 127.28.248.190:39227)
0928 02:46:41.623954 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:41.623959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:41.624891 (+ 932us) server_negotiation.cc:407] Connection header received
0928 02:46:41.626353 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:41.626356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:41.626415 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:41.626519 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:41.627771 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.628301 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.629605 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.629940 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.630063 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:41.632853 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:41.632856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:41.632858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:41.632883 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:41.634351 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:41.634864 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:41.634867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:41.634868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:41.634940 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:41.635223 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:41.635225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:41.635226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:41.635386 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:41.635457 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:41.635895 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 02:46:41.635988 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:46:41.636962 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:41.624202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:54847)
0928 02:46:41.624750 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:41.624793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:41.626133 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:41.626696 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:41.626703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:41.627056 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:41.627629 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:41.627643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.628425 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.628435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:41.629480 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:41.629489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:41.630122 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:41.630126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:41.630339 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:41.631066 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:41.631085 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:41.632710 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:41.634449 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:41.634453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:41.634471 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:41.634753 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:41.635015 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:41.635017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:41.635018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:41.635140 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:41.635475 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:41.635480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:41.635694 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 02:46:41.635834 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:46:41.636072 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260928 02:46:41.638033 5880 heartbeater.cc:347] Connected to a master server at 127.28.248.190:39227
I20260928 02:46:41.638258 5880 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:41.638712 5880 heartbeater.cc:511] Master 127.28.248.190:39227 requested a full tablet report, sending...
I20260928 02:46:41.639648 5421 ts_manager.cc:194] Registered new tserver with Master: 03fdeb2f1d034bfab6e11896e97f694a (127.28.248.131:32911)
I20260928 02:46:41.640210 5421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:54847
I20260928 02:46:41.643322 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab.
Entry for principal oryx/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab.
WARNING: no policy specified for oryx/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab.
Entry for principal oryx/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab.
WARNING: no policy specified for oryx/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab.
Entry for principal oryx/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab.
I20260928 02:46:41.719467 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5483
I20260928 02:46:41.726814 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5617
I20260928 02:46:41.732069 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5751
I20260928 02:46:41.737428 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5389
I20260928 02:46:41.743572 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39227
--webserver_interface=127.28.248.190
--webserver_port=42093
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:41.851521 5889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:41.851814 5889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:41.851897 5889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:41.855822 5889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:41.855921 5889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:41.855959 5889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:41.855995 5889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:41.856024 5889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:41.861204 5889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:41.862264 5889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:41.863272 5889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:41.871778 5894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:41.871797 5895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:41.871797 5897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:41.872172 5889 server_base.cc:1034] running on GCE node
I20260928 02:46:41.872754 5889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:41.873864 5889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:41.875025 5889 hybrid_clock.cc:648] HybridClock initialized: now 1790563601874989 us; error 45 us; skew 500 ppm
Sep 28 02:46:41 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:41.877728 5889 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:46:41.878633 5889 webserver.cc:460] Webserver started at http://127.28.248.190:42093/ using document root <none> and password file <none>
I20260928 02:46:41.879189 5889 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:41.879292 5889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:41.883006 5889 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 02:46:41.885165 5904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:41.886385 5889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:41.886523 5889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "65a0b770a1164ae5bacfdf0388396ceb"
format_stamp: "Formatted at 2026-09-28 02:46:40 on dist-test-slave-r238"
server_key: "14837d3198b9c21579a933be4a00fe89"
server_key_iv: "c4ab05e1f655dca6a51751fc902e4250"
server_key_version: "encryptionkey@0"
I20260928 02:46:41.887084 5889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:41.907279 5889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:41.910486 5889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:41.910706 5889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:41.917469 5889 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:39227
I20260928 02:46:41.917527 5956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:39227 every 8 connection(s)
I20260928 02:46:41.918551 5889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:41.919317 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5889
I20260928 02:46:41.919730 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:45115
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:39227
--webserver_port=41365
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:41.923414 5957 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:41.925155 5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Bootstrap starting.
I20260928 02:46:41.941293 5957 log.cc:824] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:41.947402 5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:41.947832 5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Bootstrap complete.
I20260928 02:46:41.951924 5957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:41.952337 5957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a0b770a1164ae5bacfdf0388396ceb, State: Initialized, Role: FOLLOWER
I20260928 02:46:41.952859 5957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:41.953001 5957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:41.953092 5957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:41.953207 5957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:46:41.955027 5957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:41.955343 5957 leader_election.cc:302] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a0b770a1164ae5bacfdf0388396ceb; no voters:
I20260928 02:46:41.955679 5957 leader_election.cc:288] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:46:41.955763 5961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:46:41.956009 5961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [term 2 LEADER]: Becoming Leader. State: Replica: 65a0b770a1164ae5bacfdf0388396ceb, State: Running, Role: LEADER
I20260928 02:46:41.956344 5961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } }
I20260928 02:46:41.956878 5957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:41.957666 5962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65a0b770a1164ae5bacfdf0388396ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } } }
I20260928 02:46:41.957805 5962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:41.958176 5963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a0b770a1164ae5bacfdf0388396ceb. Latest consensus state: current_term: 2 leader_uuid: "65a0b770a1164ae5bacfdf0388396ceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a0b770a1164ae5bacfdf0388396ceb" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 39227 } } }
I20260928 02:46:41.958304 5963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:41.958667 5970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:41.961728 5970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:41.962314 5970 catalog_manager.cc:1259] Loaded cluster ID: 5406fcf369e042f0a3a24e96c925798e
I20260928 02:46:41.962373 5970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:41.965133 5970 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:41.965749 5970 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65a0b770a1164ae5bacfdf0388396ceb: Loaded TSK: 0
I20260928 02:46:41.966912 5970 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:46:42.034155 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:42.034410 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:42.034452 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:42.038395 5959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:42.038450 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:42.038545 5959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:42.043581 5959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.129.keytab
--principal=oryx/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:45115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=41365
--tserver_master_addrs=127.28.248.190:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:42.044559 5959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:42.045562 5959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:42.056289 5983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:42.056303 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:42.056293 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:42.057255 5959 server_base.cc:1034] running on GCE node
I20260928 02:46:42.057696 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:42.058526 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:42.059749 5959 hybrid_clock.cc:648] HybridClock initialized: now 1790563602059721 us; error 43 us; skew 500 ppm
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563602, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:42.063005 5959 init.cc:377] Logged in from keytab as oryx/127.28.248.129@KRBTEST.COM (short username oryx)
I20260928 02:46:42.063980 5959 webserver.cc:460] Webserver started at http://127.28.248.129:41365/ using document root <none> and password file <none>
I20260928 02:46:42.064599 5959 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:42.064777 5959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:42.069311 5959 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:46:42.071695 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.073204 5959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:42.073354 5959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "248983fa9a544cfd946af571c35bc946"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "44c5bd39dd67011c44864bd24de6122e"
server_key_iv: "cb41597a1508655edc797f8b270b0b7c"
server_key_version: "encryptionkey@0"
I20260928 02:46:42.073989 5959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:42.096935 5959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:42.100380 5959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:42.100690 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:42.101365 5959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:42.102280 5959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:42.102360 5959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.102437 5959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:42.102492 5959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.112068 5959 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:45115
I20260928 02:46:42.112095 6106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:45115 every 8 connection(s)
I20260928 02:46:42.113217 5959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:42.116650 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 5959
I20260928 02:46:42.117206 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:34483
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:39227
--webserver_port=34017
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.127842 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.115510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:54007)
0928 02:46:42.115990 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.116043 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.117866 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.119072 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.119080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.119549 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.120049 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.120064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.121509 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.121513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.122437 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.122447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.123294 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.123299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.123499 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.124146 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.124163 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.126817 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
W20260928 02:46:42.127835 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.115356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:54007 (local address 127.28.248.190:39227)
0928 02:46:42.115871 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.115891 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.116153 (+ 262us) server_negotiation.cc:407] Connection header received
0928 02:46:42.118307 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.118320 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.118638 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.119016 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.120237 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.121361 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.122580 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.123111 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.123292 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.127054 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:54007: BlockingRecv error: recv got EOF from 127.28.248.129:54007 (error 108)
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":130,"threads_started":1}
W20260928 02:46:42.128721 6107 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:39227 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.248.190:39227: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.134696 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.129157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:36593)
0928 02:46:42.129283 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.129298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.129373 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.129617 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.129620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.129786 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.129896 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.129902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.130651 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.130654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.131571 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.131580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.132004 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.132007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.132109 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.132588 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.132601 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.134545 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:46:42.134791 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.129271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:36593 (local address 127.28.248.190:39227)
0928 02:46:42.129379 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.129382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.129395 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:42.129451 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.129453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.129510 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.129581 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.130005 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.130552 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.131675 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.131899 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.131983 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.134671 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:36593: BlockingRecv error: recv got EOF from 127.28.248.129:36593 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.140455 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.135244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:40339)
0928 02:46:42.135478 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.135491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.135561 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.135786 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.135788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.135911 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.135996 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.136001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.136980 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.136982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.137532 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.137539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.138103 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.138105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.138164 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.138624 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.138635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.140364 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174}
W20260928 02:46:42.140640 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.135360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:40339 (local address 127.28.248.190:39227)
0928 02:46:42.135481 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.135485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.135530 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:46:42.135639 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.135641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.135682 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.135763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.136097 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.136873 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.137643 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.137983 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.138102 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.140510 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:40339: BlockingRecv error: recv got EOF from 127.28.248.129:40339 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:46:42.140909 6107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:46:42.231191 6111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:42.231446 6111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:42.231488 6111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:42.235314 6111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:42.235361 6111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:42.235415 6111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:42.240391 6111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.130.keytab
--principal=oryx/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:34483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=34017
--tserver_master_addrs=127.28.248.190:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:42.241456 6111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:42.242389 6111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:42.251865 6119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:42.251895 6116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:42.251906 6117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:42.252319 6111 server_base.cc:1034] running on GCE node
I20260928 02:46:42.252735 6111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:42.253289 6111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:42.254663 6111 hybrid_clock.cc:648] HybridClock initialized: now 1790563602254652 us; error 32 us; skew 500 ppm
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563602, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:42.257541 6111 init.cc:377] Logged in from keytab as oryx/127.28.248.130@KRBTEST.COM (short username oryx)
I20260928 02:46:42.258458 6111 webserver.cc:460] Webserver started at http://127.28.248.130:34017/ using document root <none> and password file <none>
I20260928 02:46:42.259003 6111 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:42.259106 6111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:42.262996 6111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:46:42.264997 6126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.266189 6111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:42.266336 6111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "f9a255af82444751a0c0b10a9188b9e4"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "9ab0dfe1e1f2658736bb28a814a793e1"
server_key_iv: "787c25b377651725b48ad04b5835544d"
server_key_version: "encryptionkey@0"
I20260928 02:46:42.267127 6111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:42.277102 6111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:42.280041 6111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:42.280236 6111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:42.280851 6111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:42.281618 6111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:42.281687 6111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.281751 6111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:42.281801 6111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.290617 6111 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:34483
I20260928 02:46:42.290669 6239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:34483 every 8 connection(s)
I20260928 02:46:42.291685 6111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:42.293344 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6111
I20260928 02:46:42.293817 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:32911
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:39227
--webserver_port=40085
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:46:42.303718 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.293493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:34613 (local address 127.28.248.190:39227)
0928 02:46:42.293664 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.293670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.294586 (+ 916us) server_negotiation.cc:407] Connection header received
0928 02:46:42.295951 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.295955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.296029 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.296136 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.297356 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.298271 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.299405 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.299857 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.299988 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.303554 (+ 3566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:34613: BlockingRecv error: recv got EOF from 127.28.248.130:34613 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:46:42.304134 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.293901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:34613)
0928 02:46:42.294421 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.294470 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.295734 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.296287 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.296294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.296741 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.297203 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.297215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.298399 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.298402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.299264 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.299272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.300010 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.300012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.300148 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.300832 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.300847 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.303358 (+ 2511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":148,"threads_started":1}
W20260928 02:46:42.304666 6240 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:39227 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.248.190:39227: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.310223 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.305057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:50045)
0928 02:46:42.305280 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.305292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.305361 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.305603 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.305604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.305723 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.305805 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.305809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.306787 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.306789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.307309 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.307316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.307856 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.307859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.307926 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.308396 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.308406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.310135 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260928 02:46:42.310389 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.305172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:50045 (local address 127.28.248.190:39227)
0928 02:46:42.305290 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.305294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.305333 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:46:42.305423 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.305426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.305491 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.305577 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.305903 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.306677 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.307409 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.307754 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.307849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.310277 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:50045: BlockingRecv error: recv got EOF from 127.28.248.130:50045 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.315979 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.310953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:56787)
0928 02:46:42.311094 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.311110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.311201 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.311422 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.311428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.311595 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.311677 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.311683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.312317 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.312320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.313195 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.313204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.313598 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.313601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.313706 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.314102 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.314115 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.315861 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:46:42.316063 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.311052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:56787 (local address 127.28.248.190:39227)
0928 02:46:42.311148 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.311151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.311162 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:42.311288 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.311290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.311321 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.311382 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.311777 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.312224 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.313295 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.313500 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.313573 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.315959 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:56787: BlockingRecv error: recv got EOF from 127.28.248.130:56787 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:46:42.316367 6240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:46:42.408121 6243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:42.408375 6243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:42.408456 6243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:42.412526 6243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:42.412592 6243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:42.412770 6243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:42.417981 6243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/krb5kdc/oryx_127.28.248.131.keytab
--principal=oryx/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:32911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=40085
--tserver_master_addrs=127.28.248.190:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:42.419013 6243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:42.420064 6243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:42.429700 6251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:42.429764 6243 server_base.cc:1034] running on GCE node
W20260928 02:46:42.429685 6248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:42.429702 6249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:42.430294 6243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:42.430938 6243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:42.432132 6243 hybrid_clock.cc:648] HybridClock initialized: now 1790563602432121 us; error 29 us; skew 500 ppm
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563602, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:42.434898 6243 init.cc:377] Logged in from keytab as oryx/127.28.248.131@KRBTEST.COM (short username oryx)
I20260928 02:46:42.435650 6243 webserver.cc:460] Webserver started at http://127.28.248.131:40085/ using document root <none> and password file <none>
I20260928 02:46:42.436144 6243 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:42.436209 6243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:42.439787 6243 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:42.441694 6258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.442830 6243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:42.442919 6243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "03fdeb2f1d034bfab6e11896e97f694a"
format_stamp: "Formatted at 2026-09-28 02:46:41 on dist-test-slave-r238"
server_key: "20927c48c9a7aa19687035e97ec0ed3d"
server_key_iv: "f34d61668ba6190140460a521d23ffd0"
server_key_version: "encryptionkey@0"
I20260928 02:46:42.443413 6243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:42.473404 6243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:42.476754 6243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:42.476907 6243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:42.477442 6243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:42.478314 6243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:42.478360 6243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.478400 6243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:42.478421 6243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:42.486943 6243 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:32911
I20260928 02:46:42.486961 6371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:32911 every 8 connection(s)
I20260928 02:46:42.487994 6243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:42.490132 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6243
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:46:42.499703 6109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.489765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54341 (local address 127.28.248.190:39227)
0928 02:46:42.489952 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.489958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.490850 (+ 892us) server_negotiation.cc:407] Connection header received
0928 02:46:42.492474 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.492477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.492514 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.492570 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.493858 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.494470 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.495745 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.496021 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.496094 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.499577 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:54341: BlockingRecv error: recv got EOF from 127.28.248.131:54341 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:46:42.500038 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.491141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.28.248.190:39227)
0928 02:46:42.491406 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.491412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.491431 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:46:42.491512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.491515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.491572 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.491687 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.492385 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.493210 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.494131 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.494348 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.494422 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.495937 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:42.495942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:42.495955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:42.495980 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:42.497934 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:42.498435 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:42.498440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:42.498445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:42.498563 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:42.498864 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:42.498867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:42.498868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:42.499188 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:42.499335 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:42.499669 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:46:42.499905 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260928 02:46:42.500411 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.490189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:54341)
0928 02:46:42.490704 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.490754 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.492244 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.492775 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.492784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.493217 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.493728 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.493741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.494611 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.494613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.495624 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.495633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.496166 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.496172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.496380 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.496926 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.496951 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.499463 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
W20260928 02:46:42.501173 6372 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:39227 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.248.190:39227: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.506392 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.501581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:37403)
0928 02:46:42.501690 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.501705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.501776 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.502336 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.502338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.502455 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.502581 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.502592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.503264 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.503266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.503789 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.503796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.504179 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.504182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.504246 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.504725 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.504736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.506276 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:46:42.506529 6376 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.501641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:37403 (local address 127.28.248.190:39227)
0928 02:46:42.502006 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.502009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.502019 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:46:42.502070 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.502072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.502108 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.502184 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.502700 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.503174 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.503885 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.504092 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.504163 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.506389 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:37403: BlockingRecv error: recv got EOF from 127.28.248.131:37403 (error 108)
Metrics: {"server-negotiator.queue_time_us":310}
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:42 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:42.510985 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:42.506866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:55701)
0928 02:46:42.506968 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:42.506983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:42.507053 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:42.507255 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:42.507257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:42.507369 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:42.507431 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.507435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.508052 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.508054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.508538 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:42.508545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.508948 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.508951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:42.509005 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.509366 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:42.509376 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:42.510908 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:46:42.511134 6376 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:42.506949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:55701 (local address 127.28.248.190:39227)
0928 02:46:42.507087 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:42.507090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:42.507101 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:42.507126 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:42.507128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:42.507156 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:42.507217 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:42.507520 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.507951 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.508673 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:42.508863 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:42.508933 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:42.511026 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:55701: BlockingRecv error: recv got EOF from 127.28.248.131:55701 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:46:42.511241 6372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:46:43 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:43 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:43.147472 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:43.141656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:50737)
0928 02:46:43.141910 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:43.141923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:43.142009 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:43.142271 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:43.142273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:43.142484 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:43.142617 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:43.142627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.143515 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.143520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:43.144107 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:43.144116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.144581 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.144584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:43.144709 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:43.145221 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:43.145232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:43.147270 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
W20260928 02:46:43.147516 6378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:43.141736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:50737 (local address 127.28.248.190:39227)
0928 02:46:43.141967 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:43.141971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:43.141987 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:46:43.142112 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:43.142114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:43.142158 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:43.142256 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:43.142750 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.143401 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.144229 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.144458 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.144579 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:43.147396 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:50737: BlockingRecv error: recv got EOF from 127.28.248.129:50737 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":58,"threads_started":1}
Sep 28 02:46:43 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:43 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:43.322522 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:43.317138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:58721)
0928 02:46:43.317435 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:43.317455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:43.317543 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:43.317731 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:43.317733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:43.317914 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:43.318011 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:43.318016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.318827 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.318830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:43.319397 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:43.319406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.319828 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.319830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:43.319892 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:43.320489 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:43.320500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:43.322344 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260928 02:46:43.322613 6378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:43.317345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:58721 (local address 127.28.248.190:39227)
0928 02:46:43.317487 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:43.317491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:43.317503 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:43.317610 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:43.317612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:43.317650 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:43.317718 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:43.318118 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.318637 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.319527 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.319733 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.319806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:43.322492 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:58721: BlockingRecv error: recv got EOF from 127.28.248.130:58721 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:46:43 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:43 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:43.517251 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:43.511930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:45765)
0928 02:46:43.512215 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:43.512229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:43.512324 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:43.512537 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:43.512540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:43.512731 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:43.512827 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:43.512832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.513529 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.513532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:43.514095 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:43.514101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.514495 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.514497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:43.514557 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:43.515076 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:43.515089 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:43.517073 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
W20260928 02:46:43.517297 6378 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:43.511981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:45765 (local address 127.28.248.190:39227)
0928 02:46:43.512131 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:43.512136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:43.512282 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:46:43.512389 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:43.512390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:43.512435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:43.512500 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:43.512925 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.513440 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.514191 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:43.514402 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:43.514476 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:43.517192 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:45765: BlockingRecv error: recv got EOF from 127.28.248.131:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:46:44 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:44 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:44.154327 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:44.148293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:54619)
0928 02:46:44.148554 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:44.148568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:44.148706 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:44.148968 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:44.148971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:44.149152 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:44.149247 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:44.149252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.149947 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.149949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:44.150862 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:44.150874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.151523 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.151527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:44.151619 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:44.152194 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:44.152229 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:44.154193 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
W20260928 02:46:44.154480 6382 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:44.148385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:54619 (local address 127.28.248.190:39227)
0928 02:46:44.148673 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:44.148677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:44.148692 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:44.148811 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:44.148813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:44.148855 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:44.148934 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:44.149343 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.149857 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.151015 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.151406 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.151493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:44.154318 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:54619: BlockingRecv error: recv got EOF from 127.28.248.129:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":109,"threads_started":1}
Sep 28 02:46:44 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:44 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:44.328989 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:44.323358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:36037)
0928 02:46:44.323662 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:44.323675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:44.323814 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:44.324091 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:44.324093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:44.324298 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:44.324413 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:44.324419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.325265 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.325270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:44.325859 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:44.325867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.326339 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.326342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:44.326438 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:44.326907 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:44.326921 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:44.328801 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
W20260928 02:46:44.329032 6382 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:44.323425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:36037 (local address 127.28.248.190:39227)
0928 02:46:44.323605 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:44.323610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:44.323754 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:46:44.323925 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:44.323928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:44.323975 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:44.324072 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:44.324532 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.325149 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.325987 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.326227 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.326336 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:44.328928 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:36037: BlockingRecv error: recv got EOF from 127.28.248.130:36037 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 02:46:44 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:44 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:44.523638 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:44.518122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:41531)
0928 02:46:44.518417 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:44.518430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:44.518533 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:44.518829 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:44.518832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:44.519038 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:44.519133 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:44.519138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.519930 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.519935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:44.520509 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:44.520516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.521008 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.521011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:44.521101 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:44.521623 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:44.521634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:44.523436 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
W20260928 02:46:44.523660 6382 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:44.518229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:41531 (local address 127.28.248.190:39227)
0928 02:46:44.518376 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:44.518381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:44.518506 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:46:44.518646 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:44.518650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:44.518705 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:44.518797 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:44.519255 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.519815 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.520664 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:44.520893 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:44.521003 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:44.523560 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:41531: BlockingRecv error: recv got EOF from 127.28.248.131:41531 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:46:45 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:45 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:45.161401 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:45.155241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:40897)
0928 02:46:45.155558 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:45.155578 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:45.155685 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:45.155989 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:45.155992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:45.156202 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:45.156296 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:45.156302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.157297 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.157302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:45.157919 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:45.157929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.158535 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.158539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:45.158628 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:45.159195 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:45.159206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:45.161247 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
W20260928 02:46:45.161504 6386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:45.155377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:40897 (local address 127.28.248.190:39227)
0928 02:46:45.155678 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:45.155682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:45.155695 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:45.155821 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:45.155823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:45.155866 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:45.155964 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:45.156427 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.157186 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.158094 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.158420 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.158541 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:45.161394 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:40897: BlockingRecv error: recv got EOF from 127.28.248.129:40897 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":115,"threads_started":1}
Sep 28 02:46:45 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:45 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:45.335568 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:45.329946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:38623)
0928 02:46:45.330317 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:45.330332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:45.330471 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:45.331060 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:45.331063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:45.331271 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:45.331367 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:45.331374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.332105 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.332107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:45.332698 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:45.332706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.333122 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.333125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:45.333191 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:45.333671 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:45.333683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:45.335439 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":79,"threads_started":1}
W20260928 02:46:45.335692 6386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:45.329977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:38623 (local address 127.28.248.190:39227)
0928 02:46:45.330170 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:45.330176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:45.330407 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:46:45.330710 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:45.330714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:45.330796 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:45.330883 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:45.331488 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.332011 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.332794 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.333025 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.333102 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:45.335574 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:38623: BlockingRecv error: recv got EOF from 127.28.248.130:38623 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 02:46:45 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:45 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:45.530504 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:45.524509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:38583)
0928 02:46:45.524828 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:45.524841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:45.524941 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:45.525179 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:45.525181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:45.525352 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:45.525447 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:45.525452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.526359 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.526362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:45.527276 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:45.527286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.527712 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.527714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:45.527774 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:45.528323 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:45.528335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:45.530383 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":118,"threads_started":1}
W20260928 02:46:45.530704 6386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:45.524594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:38583 (local address 127.28.248.190:39227)
0928 02:46:45.524748 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:45.524753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:45.524913 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:46:45.525021 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:45.525024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:45.525070 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:45.525141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:45.525559 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.526223 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.527398 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:45.527616 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:45.527697 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:45.530524 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:38583: BlockingRecv error: recv got EOF from 127.28.248.131:38583 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:46:46 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:46 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:46.172868 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:46.162425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:43677)
0928 02:46:46.162687 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:46.162701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:46.162789 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:46.163099 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:46.163101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:46.163273 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:46.163373 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:46.163378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.164148 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.164151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:46.164831 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:46.164850 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.165370 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.165373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:46.165437 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:46.170628 (+ 5191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:46.170645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:46.172681 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260928 02:46:46.173055 6390 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:46.162506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:43677 (local address 127.28.248.190:39227)
0928 02:46:46.162868 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:46.162872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:46.162884 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:46.162919 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:46.162921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:46.162961 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:46.163057 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:46.163479 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.164022 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.164984 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.165252 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.165360 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:46.172878 (+ 7518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:43677: BlockingRecv error: recv got EOF from 127.28.248.129:43677 (error 108)
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":211,"threads_started":1}
Sep 28 02:46:46 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:46 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:46.341651 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:46.336414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:59069)
0928 02:46:46.336748 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:46.336761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:46.336856 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:46.337086 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:46.337088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:46.337265 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:46.337359 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:46.337364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.338062 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.338064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:46.338612 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:46.338619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.338998 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.339001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:46.339061 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:46.339579 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:46.339591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:46.341528 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":98,"threads_started":1}
W20260928 02:46:46.341743 6390 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:46.336516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:59069 (local address 127.28.248.190:39227)
0928 02:46:46.336740 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:46.336744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:46.336830 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:46:46.336921 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:46.336923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:46.336966 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:46.337053 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:46.337452 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.337956 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.338704 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.338908 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.338993 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:46.341651 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:59069: BlockingRecv error: recv got EOF from 127.28.248.130:59069 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 28 02:46:46 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:46 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:46.536865 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:46.531225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:34021)
0928 02:46:46.531583 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:46.531595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:46.531723 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:46.531992 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:46.531994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:46.532194 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:46.532284 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:46.532289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.533103 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.533107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:46.533670 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:46.533677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.534134 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.534138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:46.534230 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:46.534783 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:46.534794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:46.536749 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
W20260928 02:46:46.537046 6390 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:46.531302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:34021 (local address 127.28.248.190:39227)
0928 02:46:46.531456 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:46.531462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:46.531669 (+ 207us) server_negotiation.cc:407] Connection header received
0928 02:46:46.531820 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:46.531824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:46.531877 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:46.531963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:46.532400 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.532997 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.533789 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:46.534017 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:46.534120 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:46.536908 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:34021: BlockingRecv error: recv got EOF from 127.28.248.131:34021 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:46:47 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:47 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:47.179263 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:47.173605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:32887)
0928 02:46:47.173915 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:47.173928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:47.174032 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:47.174292 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:47.174294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:47.174473 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:47.174571 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:47.174576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.175321 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.175324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:47.175879 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:47.175887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.176353 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.176357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:47.176448 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:47.177120 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:47.177135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:47.179091 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
W20260928 02:46:47.179327 6394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:47.173756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:32887 (local address 127.28.248.190:39227)
0928 02:46:47.173988 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:47.173991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:47.174004 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:47.174137 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:47.174139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:47.174181 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:47.174255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:47.174669 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.175222 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.176002 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.176234 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.176308 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:47.179209 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:32887: BlockingRecv error: recv got EOF from 127.28.248.129:32887 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":54,"threads_started":1}
Sep 28 02:46:47 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:47 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:47.348997 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:47.342589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:57965)
0928 02:46:47.342848 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:47.342861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:47.342940 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:47.343235 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:47.343237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:47.343407 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:47.343500 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:47.343505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.344783 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.344785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:47.345343 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:47.345350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.346035 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.346038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:47.346100 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:47.346721 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:47.346733 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:47.348883 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
W20260928 02:46:47.349344 6394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:47.342737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:57965 (local address 127.28.248.190:39227)
0928 02:46:47.342900 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:47.342906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:47.342924 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:46:47.343041 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:47.343047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:47.343110 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:47.343208 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:47.343626 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.344674 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.345463 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.345923 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.346050 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:47.349080 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:57965: BlockingRecv error: recv got EOF from 127.28.248.130:57965 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:46:47 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:47 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:47.542834 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:47.537634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:38997)
0928 02:46:47.537900 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:47.537913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:47.538003 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:47.538230 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:47.538232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:47.538410 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:47.538499 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:47.538504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.539227 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.539229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:47.539785 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:47.539792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.540205 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.540208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:47.540267 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:47.540821 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:47.540832 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:47.542646 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20260928 02:46:47.542865 6394 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:47.537724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:38997 (local address 127.28.248.190:39227)
0928 02:46:47.537903 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:47.537907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:47.537979 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:46:47.538080 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:47.538082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:47.538126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:47.538195 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:47.538605 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.539138 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.539886 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:47.540110 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:47.540188 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:47.542763 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:38997: BlockingRecv error: recv got EOF from 127.28.248.131:38997 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 02:46:48 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:48 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:48.185937 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:48.180030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:56299)
0928 02:46:48.180356 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:48.180371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:48.180505 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:48.180820 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:48.180823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:48.181076 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:48.181227 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:48.181234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.181988 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.181993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:48.182571 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:48.182579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.183059 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.183062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:48.183166 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:48.183842 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:48.183854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:48.185767 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
W20260928 02:46:48.186012 6398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:48.180090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:56299 (local address 127.28.248.190:39227)
0928 02:46:48.180329 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:48.180334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:48.180469 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:46:48.180637 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:48.180640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:48.180696 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:48.180796 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:48.181343 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.181877 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.182699 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.182933 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.183060 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:48.185888 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:56299: BlockingRecv error: recv got EOF from 127.28.248.129:56299 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
Sep 28 02:46:48 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:48 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:48.355615 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:48.349765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:53953)
0928 02:46:48.350060 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:48.350073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:48.350172 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:48.350409 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:48.350411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:48.350612 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:48.350703 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:48.350708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.351474 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.351476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:48.352114 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:48.352123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.352567 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.352570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:48.352698 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:48.353248 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:48.353259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:48.355502 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
W20260928 02:46:48.355818 6398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:48.349825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:53953 (local address 127.28.248.190:39227)
0928 02:46:48.349974 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:48.349979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:48.350145 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:46:48.350253 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:48.350256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:48.350301 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:48.350366 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:48.350812 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.351382 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.352249 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.352467 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.352558 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:48.355674 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:53953: BlockingRecv error: recv got EOF from 127.28.248.130:53953 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:46:48 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:48 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:48.548910 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:48.543622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:33579)
0928 02:46:48.543893 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:48.543906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:48.543999 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:48.544215 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:48.544217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:48.544396 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:48.544501 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:48.544506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.545224 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.545226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:48.545791 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:48.545798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.546190 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.546192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:48.546254 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:48.546853 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:48.546864 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:48.548793 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
W20260928 02:46:48.549091 6398 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:48.543686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:33579 (local address 127.28.248.190:39227)
0928 02:46:48.543814 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:48.543817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:48.543972 (+ 155us) server_negotiation.cc:407] Connection header received
0928 02:46:48.544063 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:48.544065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:48.544111 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:48.544175 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:48.544599 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.545136 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.545890 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:48.546098 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:48.546170 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:48.548942 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:33579: BlockingRecv error: recv got EOF from 127.28.248.131:33579 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:46:49 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:49 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:49.192381 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:49.186728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:43741)
0928 02:46:49.187063 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:49.187076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:49.187167 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:49.187425 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:49.187427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:49.187621 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:49.187715 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:49.187720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.188437 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.188440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:49.189074 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:49.189083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.189674 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.189678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:49.189773 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:49.190448 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:49.190462 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:49.192269 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":84,"threads_started":1}
W20260928 02:46:49.192544 6402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:49.186863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:43741 (local address 127.28.248.190:39227)
0928 02:46:49.187133 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:49.187136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:49.187149 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:49.187265 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:49.187267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:49.187311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:49.187386 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:49.187817 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.188349 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.189183 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.189542 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.189634 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:49.192395 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:43741: BlockingRecv error: recv got EOF from 127.28.248.129:43741 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":61,"threads_started":1}
Sep 28 02:46:49 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:49 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:49.362541 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:49.356443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:55573)
0928 02:46:49.356778 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:49.356791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:49.356889 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:49.357125 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:49.357127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:49.357366 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:49.357501 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:49.357507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.358437 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.358440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:49.359077 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:49.359086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.359574 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.359576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:49.359638 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:49.360122 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:49.360133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:49.362364 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
W20260928 02:46:49.362630 6402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:49.356549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:55573 (local address 127.28.248.190:39227)
0928 02:46:49.356746 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:49.356749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:49.356915 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:46:49.356961 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:49.356963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:49.357013 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:49.357092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:49.357658 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.358304 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.359200 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.359456 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.359551 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:49.362500 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:55573: BlockingRecv error: recv got EOF from 127.28.248.130:55573 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 02:46:49 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:49 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:49.554879 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:49.549732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:58293)
0928 02:46:49.550028 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:49.550041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:49.550138 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:49.550369 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:49.550371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:49.550560 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:49.550663 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:49.550668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.551370 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.551372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:49.551927 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:49.551933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.552338 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.552340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:49.552403 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:49.552911 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:49.552922 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:49.554769 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":66,"threads_started":1}
W20260928 02:46:49.555063 6402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:49.549834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:58293 (local address 127.28.248.190:39227)
0928 02:46:49.550021 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:49.550025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:49.550117 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:46:49.550223 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:49.550225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:49.550267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:49.550333 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:49.550760 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.551282 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.552029 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:49.552245 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:49.552320 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:49.554896 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:58293: BlockingRecv error: recv got EOF from 127.28.248.131:58293 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 02:46:50 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:50 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:50.198467 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:50.193141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:52513)
0928 02:46:50.193465 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:50.193478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:50.193564 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:50.193848 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:50.193850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:50.194027 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:50.194144 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:50.194150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.194908 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.194911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:50.195471 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:50.195479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.195889 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.195891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:50.195950 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:50.196431 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:50.196442 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:50.198301 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
W20260928 02:46:50.198537 6406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:50.193299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:52513 (local address 127.28.248.190:39227)
0928 02:46:50.193554 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:50.193557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:50.193572 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:50.193671 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:50.193673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:50.193718 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:50.193791 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:50.194271 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.194808 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.195582 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.195799 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.195872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:50.198412 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:52513: BlockingRecv error: recv got EOF from 127.28.248.129:52513 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":59,"threads_started":1}
Sep 28 02:46:50 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:50 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:50.369431 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:50.363425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:33173)
0928 02:46:50.363761 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:50.363774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:50.363872 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:50.364227 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:50.364230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:50.364491 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:50.364677 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:50.364687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.365710 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.365713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:50.366329 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:50.366337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.366890 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.366893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:50.366968 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:50.367429 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:50.367443 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:50.369314 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
W20260928 02:46:50.369596 6406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:50.363548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:33173 (local address 127.28.248.190:39227)
0928 02:46:50.363722 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:50.363726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:50.363845 (+ 119us) server_negotiation.cc:407] Connection header received
0928 02:46:50.364001 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:50.364004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:50.364078 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:50.364162 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:50.364790 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.365588 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.366472 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.366792 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.366870 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:50.369446 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:33173: BlockingRecv error: recv got EOF from 127.28.248.130:33173 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 02:46:50 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:50 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:50.561726 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:50.555667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:47773)
0928 02:46:50.555997 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:50.556013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:50.556131 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:50.556413 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:50.556415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:50.556721 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:50.556817 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:50.556823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.557655 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.557660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:50.558343 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:50.558352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.558803 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.558806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:50.558895 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:50.559513 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:50.559528 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:50.561555 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
W20260928 02:46:50.561787 6406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:50.555868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:47773 (local address 127.28.248.190:39227)
0928 02:46:50.556074 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:50.556078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:50.556092 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:50.556253 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:50.556262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:50.556306 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:50.556392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:50.556932 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.557540 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.558456 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:50.558690 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:50.558799 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:50.561682 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:47773: BlockingRecv error: recv got EOF from 127.28.248.131:47773 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 02:46:51 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:51 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:51.210428 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:51.199253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:33301)
0928 02:46:51.199523 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:51.199537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:51.199625 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:51.199857 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:51.199859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:51.200037 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:51.200135 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:51.200140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.200880 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.200882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:51.201439 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:51.201447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.201852 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.201854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:51.201915 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:51.208413 (+ 6498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:51.208425 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:51.210313 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
W20260928 02:46:51.210608 6410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:51.199306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:33301 (local address 127.28.248.190:39227)
0928 02:46:51.199547 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:51.199562 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:51.199580 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:46:51.199702 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:51.199705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:51.199745 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:51.199822 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:51.200233 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.200788 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.201539 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.201762 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.201834 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:51.210449 (+ 8615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:33301: BlockingRecv error: recv got EOF from 127.28.248.129:33301 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
Sep 28 02:46:51 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:51 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:51.375542 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:51.370182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:60205)
0928 02:46:51.370484 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:51.370499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:51.370612 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:51.370868 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:51.370870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:51.371068 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:51.371182 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:51.371188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.371993 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.371996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:51.372567 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:51.372575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.373041 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.373043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:51.373103 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:51.373577 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:51.373588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:51.375406 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":140,"threads_started":1}
W20260928 02:46:51.375672 6410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:51.370235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:60205 (local address 127.28.248.190:39227)
0928 02:46:51.370410 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:51.370415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:51.370561 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:46:51.370697 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:51.370700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:51.370764 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:51.370833 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:51.371281 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.371900 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.372726 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.372933 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.373006 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:51.375538 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:60205: BlockingRecv error: recv got EOF from 127.28.248.130:60205 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 02:46:51 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:51 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:51.569239 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:51.562593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:42959)
0928 02:46:51.562900 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:51.562917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:51.563028 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:51.563289 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:51.563292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:51.563552 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:51.563704 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:51.563711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.564530 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.564533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:51.565245 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:51.565253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.565922 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.565925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:51.566031 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:51.566765 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:51.566785 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:51.569032 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
W20260928 02:46:51.569358 6410 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:51.562947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:42959 (local address 127.28.248.190:39227)
0928 02:46:51.563066 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:51.563070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:51.563084 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:51.563122 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:51.563124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:51.563176 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:51.563258 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:51.563825 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.564439 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.565348 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:51.565796 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:51.565908 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:51.569201 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:42959: BlockingRecv error: recv got EOF from 127.28.248.131:42959 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 02:46:52 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:52 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:52.216483 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:52.211266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:41131)
0928 02:46:52.211536 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:52.211550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:52.211644 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:52.211893 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:52.211895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:52.212071 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:52.212169 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:52.212174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.212914 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.212916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:52.213467 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:52.213476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.213888 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.213891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:52.213953 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:52.214428 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:52.214439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:52.216308 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260928 02:46:52.216548 6414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:52.211340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:41131 (local address 127.28.248.190:39227)
0928 02:46:52.211567 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:52.211571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:52.211600 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:46:52.211726 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:52.211728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:52.211770 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:52.211845 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:52.212265 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.212811 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.213576 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.213786 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.213857 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:52.216426 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:41131: BlockingRecv error: recv got EOF from 127.28.248.129:41131 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
Sep 28 02:46:52 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:52 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:52.382205 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:52.376465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:57547)
0928 02:46:52.376769 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:52.376783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:52.376883 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:52.377139 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:52.377141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:52.377379 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:52.377496 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:52.377503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.378420 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.378425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:52.379065 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:52.379074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.379523 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.379527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:52.379619 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:52.380179 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:52.380191 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:52.382037 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
W20260928 02:46:52.382277 6414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:52.376578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:57547 (local address 127.28.248.190:39227)
0928 02:46:52.376801 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:52.376805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:52.376839 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:52.376979 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:52.376981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:52.377031 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:52.377101 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:52.377648 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.378312 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.379184 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.379409 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.379519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:52.382168 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:57547: BlockingRecv error: recv got EOF from 127.28.248.130:57547 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 28 02:46:52 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:52 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:52.575901 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:52.570089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:53893)
0928 02:46:52.570474 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:52.570487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:52.570600 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:52.570878 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:52.570881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:52.571066 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:52.571163 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:52.571167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.572012 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.572015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:52.572783 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:52.572793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.573249 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.573253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:52.573336 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:52.573888 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:52.573900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:52.575787 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260928 02:46:52.576078 6414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:52.570267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:53893 (local address 127.28.248.190:39227)
0928 02:46:52.570422 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:52.570426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:52.570559 (+ 133us) server_negotiation.cc:407] Connection header received
0928 02:46:52.570687 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:52.570689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:52.570756 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:52.570833 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:52.571288 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.571910 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.572903 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:52.573135 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:52.573246 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:52.575920 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:53893: BlockingRecv error: recv got EOF from 127.28.248.131:53893 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:46:53 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:53 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:53.222838 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:53.217305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:40487)
0928 02:46:53.217580 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:53.217596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:53.217701 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:53.217960 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:53.217963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:53.218141 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:53.218242 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:53.218247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.219050 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.219054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:53.219636 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:53.219645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.220075 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.220078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:53.220148 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:53.220734 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:53.220747 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:53.222723 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
W20260928 02:46:53.223021 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:53.217428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:40487 (local address 127.28.248.190:39227)
0928 02:46:53.217674 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:53.217678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:53.217691 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:53.217787 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:53.217789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:53.217831 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:53.217907 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:53.218352 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.218888 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.219749 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.219967 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.220044 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:53.222850 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:40487: BlockingRecv error: recv got EOF from 127.28.248.129:40487 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
Sep 28 02:46:53 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:53 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:53.388857 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:53.382978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:58153)
0928 02:46:53.383262 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:53.383277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:53.383409 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:53.383701 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:53.383704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:53.383920 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:53.384015 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:53.384021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.384841 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.384846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:53.385418 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:53.385426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.385881 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.385884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:53.385972 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:53.386523 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:53.386545 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:53.388688 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
W20260928 02:46:53.388898 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:53.383084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:58153 (local address 127.28.248.190:39227)
0928 02:46:53.383237 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:53.383242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:53.383357 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:46:53.383515 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:53.383520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:53.383587 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:53.383676 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:53.384137 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.384734 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.385540 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.385769 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.385876 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:53.388802 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:58153: BlockingRecv error: recv got EOF from 127.28.248.130:58153 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:46:53 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:53 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:53.582232 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:53.576668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:53229)
0928 02:46:53.576929 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:53.576943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:53.577037 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:53.577265 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:53.577267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:53.577495 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:53.577605 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:53.577612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.578350 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.578352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:53.578970 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:53.578978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.579555 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.579579 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:53.579670 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:53.580314 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:53.580328 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:53.582123 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
W20260928 02:46:53.582362 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:53.576740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:53229 (local address 127.28.248.190:39227)
0928 02:46:53.576871 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:53.576874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:53.577004 (+ 130us) server_negotiation.cc:407] Connection header received
0928 02:46:53.577115 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:53.577117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:53.577160 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:53.577228 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:53.577703 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.578260 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.579108 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:53.579428 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:53.579538 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:53.582247 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:53229: BlockingRecv error: recv got EOF from 127.28.248.131:53229 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:46:54 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:54 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:54.228717 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:54.223575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:48345)
0928 02:46:54.223857 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:54.223871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:54.223972 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:54.224230 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:54.224232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:54.224407 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:54.224502 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:54.224507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.225253 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.225255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:54.225809 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:54.225815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.226245 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.226247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:54.226309 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:54.226774 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:54.226785 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:54.228546 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
W20260928 02:46:54.228899 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:54.223627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:48345 (local address 127.28.248.190:39227)
0928 02:46:54.223859 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:54.223862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:54.223942 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:46:54.224067 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:54.224069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:54.224113 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:54.224187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:54.224596 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.225162 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.225906 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.226149 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.226225 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:54.228729 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:48345: BlockingRecv error: recv got EOF from 127.28.248.129:48345 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
Sep 28 02:46:54 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:54 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:54.395071 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:54.389679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:43919)
0928 02:46:54.389942 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:54.389984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:54.390103 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:54.390381 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:54.390384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:54.390603 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:54.390697 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:54.390702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.391462 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.391467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:54.392042 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:54.392049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.392514 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.392518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:54.392599 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:54.393099 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:54.393110 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:54.394894 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
W20260928 02:46:54.395109 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:54.389774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:43919 (local address 127.28.248.190:39227)
0928 02:46:54.389968 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:54.389974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:54.390057 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:46:54.390208 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:54.390211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:54.390268 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:54.390350 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:54.390810 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.391353 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.392159 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.392403 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.392520 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:54.395012 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:43919: BlockingRecv error: recv got EOF from 127.28.248.130:43919 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:46:54 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:54 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:54.588449 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:54.582967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:52865)
0928 02:46:54.583249 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:54.583262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:54.583347 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:54.583584 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:54.583586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:54.583763 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:54.583856 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:54.583860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.584647 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.584650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:54.585221 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:54.585228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.585649 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.585651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:54.585769 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:54.586375 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:54.586409 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:54.588336 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
W20260928 02:46:54.588580 6422 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:54.583164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:52865 (local address 127.28.248.190:39227)
0928 02:46:54.583295 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:54.583298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:54.583312 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:54.583430 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:54.583433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:54.583476 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:54.583547 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:54.583983 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.584514 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.585329 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:54.585552 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:54.585633 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:54.588465 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:52865: BlockingRecv error: recv got EOF from 127.28.248.131:52865 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 02:46:55 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:55 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:55.235291 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.229427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:41925)
0928 02:46:55.229695 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.229708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.229794 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.230039 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.230041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.230218 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.230312 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.230317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.231138 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.231140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.231695 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.231703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.232221 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.232223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.232287 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.232927 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.232939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.235084 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
W20260928 02:46:55.235360 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:55.229559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:41925 (local address 127.28.248.190:39227)
0928 02:46:55.229784 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.229787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.229801 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:55.229883 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.229885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.229927 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.229998 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.230421 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.231033 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.231811 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.232115 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.232202 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.235220 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:41925: BlockingRecv error: recv got EOF from 127.28.248.129:41925 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
Sep 28 02:46:55 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:55 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:55.401923 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.395800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:38455)
0928 02:46:55.396070 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.396086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.396196 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.396491 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.396497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.396777 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.396899 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.396914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.397985 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.397988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.398568 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.398576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.399055 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.399057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.399117 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.399658 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.399672 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.401712 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
W20260928 02:46:55.401945 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:55.396025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:38455 (local address 127.28.248.190:39227)
0928 02:46:55.396159 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.396162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.396194 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:46:55.396292 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.396296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.396353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.396434 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.397071 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.397850 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.398683 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.398943 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.399028 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.401835 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:38455: BlockingRecv error: recv got EOF from 127.28.248.130:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:46:55 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:55 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:55.595394 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.589344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:53631)
0928 02:46:55.589649 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.589664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.589779 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.590022 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.590024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.590204 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.590321 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.590327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.591249 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.591255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.591841 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.591849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.592308 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.592312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.592400 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.592985 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.592997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.595216 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
W20260928 02:46:55.595498 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:55.589417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:53631 (local address 127.28.248.190:39227)
0928 02:46:55.589565 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.589570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.589778 (+ 208us) server_negotiation.cc:407] Connection header received
0928 02:46:55.589851 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.589854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.589903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.589986 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.590573 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.591115 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.591967 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.592195 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.592306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.595384 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:53631: BlockingRecv error: recv got EOF from 127.28.248.131:53631 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:46:56 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:56 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:56.248050 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:56.236334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:34731)
0928 02:46:56.236664 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:56.236686 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:56.236772 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:56.237073 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:56.237075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:56.237251 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:56.237346 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:56.237351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.238178 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.238184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:56.238750 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:56.238759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.239270 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.239274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:56.239346 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:56.245544 (+ 6198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:56.245563 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:56.247918 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
W20260928 02:46:56.248226 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:56.236486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:34731 (local address 127.28.248.190:39227)
0928 02:46:56.236796 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:56.236799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:56.236831 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:46:56.236867 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:56.236869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:56.236955 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:56.237029 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:56.237484 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.238042 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.238892 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.239139 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.239234 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:56.248071 (+ 8837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:34731: BlockingRecv error: recv got EOF from 127.28.248.129:34731 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
Sep 28 02:46:56 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:56 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:56.408545 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:56.402786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:60927)
0928 02:46:56.403128 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:56.403145 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:56.403293 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:56.403588 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:56.403591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:56.403810 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:56.403900 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:56.403906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.404763 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.404768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:56.405351 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:56.405360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.405819 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.405823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:56.405911 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:56.406490 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:56.406502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:56.408409 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":89,"threads_started":1}
W20260928 02:46:56.408805 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:56.402960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:60927 (local address 127.28.248.190:39227)
0928 02:46:56.403127 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:56.403131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:56.403235 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:46:56.403395 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:56.403399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:56.403451 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:56.403590 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:56.404068 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.404607 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.405475 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.405709 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.405819 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:56.408553 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:60927: BlockingRecv error: recv got EOF from 127.28.248.130:60927 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:46:56 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:56 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.131@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:56.601394 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:56.596242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.131:57373)
0928 02:46:56.596502 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:56.596515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:56.596647 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:56.596867 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:56.596869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:56.597052 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:56.597148 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:56.597153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.597867 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.597870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:56.598452 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:56.598460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.598863 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.598865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:56.598926 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:56.599412 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:56.599424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:56.601269 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":102,"threads_started":1}
W20260928 02:46:56.601507 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:56.596278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:57373 (local address 127.28.248.190:39227)
0928 02:46:56.596422 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:56.596425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:56.596640 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:46:56.596717 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:56.596719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:56.596765 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:56.596833 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:56.597241 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.597762 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.598551 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:56.598773 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:56.598846 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:56.601393 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.131:57373: BlockingRecv error: recv got EOF from 127.28.248.131:57373 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:46:57 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:57 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.129@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:57.255198 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:57.249394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.129:52517)
0928 02:46:57.249711 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:57.249725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:57.249816 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:57.250202 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:57.250204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:57.250381 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:57.250483 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:57.250488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.251359 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.251365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:57.251955 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:57.251964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.252441 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.252445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:57.252528 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.253061 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:57.253073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:57.255065 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
W20260928 02:46:57.255426 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:57.249463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:52517 (local address 127.28.248.190:39227)
0928 02:46:57.249750 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:57.249753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:57.249771 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:46:57.250019 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:57.250022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:57.250074 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:57.250169 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:57.250631 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.251229 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.252096 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.252322 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.252430 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.255239 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.129:52517: BlockingRecv error: recv got EOF from 127.28.248.129:52517 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
Sep 28 02:46:57 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:46:57 dist-test-slave-r238 krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.248.130@KRBTEST.COM for oryx/127.28.248.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:46:57.415335 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:57.409461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:39227 (local address 127.28.248.130:55315)
0928 02:46:57.409742 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:57.409763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:57.409882 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:57.410185 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:57.410187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:57.410402 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:57.410494 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:57.410500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.411309 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.411312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:57.411883 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:57.411890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.412345 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.412348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:57.412434 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.412986 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:57.412998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:57.415142 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.248.190:39227: Server oryx/127.28.248.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20260928 02:46:57.415380 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:57.409524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:55315 (local address 127.28.248.190:39227)
0928 02:46:57.409701 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:57.409706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:57.409832 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:46:57.409992 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:57.409995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:57.410066 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:57.410157 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:57.410650 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.411189 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.412010 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.412235 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.412343 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.415279 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.248.130:55315: BlockingRecv error: recv got EOF from 127.28.248.130:55315 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:46:57.494056 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5959
I20260928 02:46:57.499841 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6111
I20260928 02:46:57.505091 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6243
I20260928 02:46:57.510427 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 5889
2026-09-28T02:46:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18008 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:57Z Disabled control of system clock
I20260928 02:46:57.540211 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41507
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:41507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:46:57.651610 6439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:57.651882 6439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:57.651923 6439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:57.656149 6439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:57.656224 6439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:57.656239 6439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:57.656256 6439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:57.656271 6439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:57.661271 6439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:41507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:57.662356 6439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:57.663730 6439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:57.672756 6447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:57.672744 6444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:57.672856 6445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:57.672880 6439 server_base.cc:1034] running on GCE node
I20260928 02:46:57.673681 6439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:57.674845 6439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:57.676033 6439 hybrid_clock.cc:648] HybridClock initialized: now 1790563617676000 us; error 47 us; skew 500 ppm
I20260928 02:46:57.677878 6439 webserver.cc:460] Webserver started at http://127.28.248.190:44323/ using document root <none> and password file <none>
I20260928 02:46:57.678457 6439 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:57.678550 6439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:57.678785 6439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:57.680871 6439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "695d49d151414d47b7491d08158b1349"
format_stamp: "Formatted at 2026-09-28 02:46:57 on dist-test-slave-r238"
server_key: "a7912c15a8218d581fc4896bd62e5494"
server_key_iv: "bb33c50673d150286746139c58e7acbb"
server_key_version: "encryptionkey@0"
I20260928 02:46:57.681448 6439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "695d49d151414d47b7491d08158b1349"
format_stamp: "Formatted at 2026-09-28 02:46:57 on dist-test-slave-r238"
server_key: "a7912c15a8218d581fc4896bd62e5494"
server_key_iv: "bb33c50673d150286746139c58e7acbb"
server_key_version: "encryptionkey@0"
I20260928 02:46:57.685091 6439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:46:57.687453 6453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:57.688714 6439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:57.688844 6439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "695d49d151414d47b7491d08158b1349"
format_stamp: "Formatted at 2026-09-28 02:46:57 on dist-test-slave-r238"
server_key: "a7912c15a8218d581fc4896bd62e5494"
server_key_iv: "bb33c50673d150286746139c58e7acbb"
server_key_version: "encryptionkey@0"
I20260928 02:46:57.688951 6439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:57.706327 6439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:57.707036 6439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:57.707239 6439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:57.713832 6439 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:41507
I20260928 02:46:57.713841 6505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:41507 every 8 connection(s)
I20260928 02:46:57.714766 6439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:57.715883 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6439
I20260928 02:46:57.716004 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:57.716286 29666 external_mini_cluster.cc:1428] Setting key 8dbb063f820ba77235eea341fc047ebe
I20260928 02:46:57.718434 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:57.723820 6506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349: Bootstrap starting.
I20260928 02:46:57.725430 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:57.717687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.28.248.190:41507)
0928 02:46:57.718347 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:57.718361 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:57.718396 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:46:57.719085 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:57.719101 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:57.719480 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:57.719819 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:57.720230 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.721248 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.722116 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.722563 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.722726 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.723625 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:57.723630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:57.723643 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:57.724103 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:57.724169 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:57.724209 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:57.724321 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:57.724462 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:46:57.724742 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":158,"threads_started":1}
I20260928 02:46:57.726122 6506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:57.727012 6506 log.cc:824] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:57.729007 6506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349: No bootstrap required, opened a new log
I20260928 02:46:57.732712 6506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d49d151414d47b7491d08158b1349" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41507 } }
I20260928 02:46:57.733001 6506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:57.733052 6506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695d49d151414d47b7491d08158b1349, State: Initialized, Role: FOLLOWER
I20260928 02:46:57.733457 6506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d49d151414d47b7491d08158b1349" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41507 } }
I20260928 02:46:57.733582 6506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:57.733623 6506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:57.733693 6506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:57.734539 6506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d49d151414d47b7491d08158b1349" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41507 } }
I20260928 02:46:57.734824 6506 leader_election.cc:302] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695d49d151414d47b7491d08158b1349; no voters:
I20260928 02:46:57.735119 6506 leader_election.cc:288] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:57.735237 6511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:57.735452 6511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [term 1 LEADER]: Becoming Leader. State: Replica: 695d49d151414d47b7491d08158b1349, State: Running, Role: LEADER
I20260928 02:46:57.735785 6511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d49d151414d47b7491d08158b1349" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41507 } }
I20260928 02:46:57.736467 6506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:57.737501 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695d49d151414d47b7491d08158b1349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d49d151414d47b7491d08158b1349" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41507 } } }
I20260928 02:46:57.737651 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:57.738025 6520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:57.738021 6513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695d49d151414d47b7491d08158b1349. Latest consensus state: current_term: 1 leader_uuid: "695d49d151414d47b7491d08158b1349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695d49d151414d47b7491d08158b1349" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41507 } } }
I20260928 02:46:57.738643 6513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:57.740993 6520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:57.746403 6520 catalog_manager.cc:1347] Generated new cluster ID: ce0f85558c3e4b96935c30a0f924bca2
I20260928 02:46:57.746469 6520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:57.769263 6520 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:57.770572 6520 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:57.779672 6520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695d49d151414d47b7491d08158b1349: Generated new TSK 0
I20260928 02:46:57.780525 6520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:57.793941 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41507
--builtin_ntp_servers=127.28.248.148:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:46:57.907445 6530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:57.907716 6530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:57.907752 6530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:57.911782 6530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:57.911842 6530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:57.911895 6530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:57.916486 6530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:57.917547 6530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:57.918866 6530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:57.928452 6538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:57.928566 6530 server_base.cc:1034] running on GCE node
W20260928 02:46:57.928460 6535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:57.928762 6536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:57.929155 6530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:57.929780 6530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:57.930941 6530 hybrid_clock.cc:648] HybridClock initialized: now 1790563617930916 us; error 37 us; skew 500 ppm
I20260928 02:46:57.932756 6530 webserver.cc:460] Webserver started at http://127.28.248.129:35725/ using document root <none> and password file <none>
I20260928 02:46:57.933295 6530 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:57.933350 6530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:57.933516 6530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:57.935482 6530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "a9ba59afdb60474491bffc136eb445c9"
format_stamp: "Formatted at 2026-09-28 02:46:57 on dist-test-slave-r238"
server_key: "c8d123e04347ac0dc1f7e50792c62a65"
server_key_iv: "8ae4a4da24be8f9831dbeb2ca56ebe90"
server_key_version: "encryptionkey@0"
I20260928 02:46:57.935972 6530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "a9ba59afdb60474491bffc136eb445c9"
format_stamp: "Formatted at 2026-09-28 02:46:57 on dist-test-slave-r238"
server_key: "c8d123e04347ac0dc1f7e50792c62a65"
server_key_iv: "8ae4a4da24be8f9831dbeb2ca56ebe90"
server_key_version: "encryptionkey@0"
I20260928 02:46:57.940384 6530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260928 02:46:57.942971 6544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:57.944154 6530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:57.944283 6530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "a9ba59afdb60474491bffc136eb445c9"
format_stamp: "Formatted at 2026-09-28 02:46:57 on dist-test-slave-r238"
server_key: "c8d123e04347ac0dc1f7e50792c62a65"
server_key_iv: "8ae4a4da24be8f9831dbeb2ca56ebe90"
server_key_version: "encryptionkey@0"
I20260928 02:46:57.944398 6530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:57.961939 6530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:57.962661 6530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:57.962872 6530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:57.963711 6530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:57.964951 6530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:57.965042 6530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:57.965145 6530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:57.965178 6530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:57.974395 6530 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:35263
I20260928 02:46:57.974412 6657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:35263 every 8 connection(s)
I20260928 02:46:57.975378 6530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:57.980504 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6530
I20260928 02:46:57.980638 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:57.980923 29666 external_mini_cluster.cc:1428] Setting key e2fb09ca696d8627ebddcf2db8ec004f
I20260928 02:46:57.982929 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41507
--builtin_ntp_servers=127.28.248.148:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:46:57.985437 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:57.977346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:36327 (local address 127.28.248.190:41507)
0928 02:46:57.977484 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:57.977488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:57.978593 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 02:46:57.980028 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:57.980032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:57.980095 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:57.980209 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:57.981219 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.981904 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.983325 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.983559 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.983673 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.984907 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:57.984910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:57.984912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:57.984925 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:57.984945 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:57.984953 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:57.985019 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:57.985250 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:46:57.985321 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:46:57.986351 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:57.977749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41507 (local address 127.28.248.129:36327)
0928 02:46:57.978414 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:57.978465 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:57.979808 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:57.980403 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:57.980410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:57.980505 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:57.981081 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:57.981095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.982047 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.982055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:57.983073 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:57.983109 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:57.983727 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:57.983731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:57.983908 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:57.984566 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:57.984570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:57.984653 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:57.984744 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:57.985044 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:57.985054 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:57.985062 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:57.985211 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 02:46:57.985539 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260928 02:46:57.987697 6658 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41507
I20260928 02:46:57.987974 6658 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:57.988706 6658 heartbeater.cc:511] Master 127.28.248.190:41507 requested a full tablet report, sending...
I20260928 02:46:57.990387 6470 ts_manager.cc:194] Registered new tserver with Master: a9ba59afdb60474491bffc136eb445c9 (127.28.248.129:35263)
I20260928 02:46:57.991571 6470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.129:36327
W20260928 02:46:58.095301 6661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:58.095562 6661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:58.095599 6661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:58.099671 6661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:58.099731 6661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:58.099784 6661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:58.104593 6661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:58.105674 6661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:58.107028 6661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:58.116819 6669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.116942 6666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.116819 6667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.117062 6661 server_base.cc:1034] running on GCE node
I20260928 02:46:58.117512 6661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:58.118199 6661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:58.119382 6661 hybrid_clock.cc:648] HybridClock initialized: now 1790563618119368 us; error 24 us; skew 500 ppm
I20260928 02:46:58.121417 6661 webserver.cc:460] Webserver started at http://127.28.248.130:39175/ using document root <none> and password file <none>
I20260928 02:46:58.122004 6661 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:58.122066 6661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:58.122327 6661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:58.124325 6661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "81f03d5cef884d2e8b9502cecf0e8a83"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "4787d26b482d62e53deed25e09e1108e"
server_key_iv: "0a94dfb62729b78c672f54c308ce9e56"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.124907 6661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "81f03d5cef884d2e8b9502cecf0e8a83"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "4787d26b482d62e53deed25e09e1108e"
server_key_iv: "0a94dfb62729b78c672f54c308ce9e56"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.128428 6661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:46:58.130867 6675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.132054 6661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:58.132191 6661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "81f03d5cef884d2e8b9502cecf0e8a83"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "4787d26b482d62e53deed25e09e1108e"
server_key_iv: "0a94dfb62729b78c672f54c308ce9e56"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.132300 6661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:58.157028 6661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:58.157766 6661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:58.157984 6661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:58.158587 6661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:58.159564 6661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:58.159636 6661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.159703 6661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:58.159751 6661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.168964 6661 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:37587
I20260928 02:46:58.168988 6788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:37587 every 8 connection(s)
I20260928 02:46:58.169948 6661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:58.171274 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6661
I20260928 02:46:58.171415 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:58.171746 29666 external_mini_cluster.cc:1428] Setting key 6dadf841620748cf17c4f87423cb3aa4
I20260928 02:46:58.173709 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41507
--builtin_ntp_servers=127.28.248.148:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:46:58.180709 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.171884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:52799 (local address 127.28.248.190:41507)
0928 02:46:58.172045 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:58.172051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:58.173285 (+ 1234us) server_negotiation.cc:407] Connection header received
0928 02:46:58.174742 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:58.174748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:58.174813 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:58.174913 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:58.176160 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.176739 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.178577 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.178783 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.178862 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:58.180202 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:58.180205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:58.180207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:58.180219 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:58.180237 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:58.180245 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:58.180296 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:58.180492 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:46:58.180542 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:46:58.182018 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.172369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41507 (local address 127.28.248.130:52799)
0928 02:46:58.173126 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:58.173178 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:58.174536 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:58.175138 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:58.175149 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:58.175265 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:58.176008 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:58.176029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.176859 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.176866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:58.178446 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:58.178462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.178950 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.178958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:58.179197 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:58.179856 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:58.179860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:58.179923 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:58.180030 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:58.180327 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:58.180339 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:58.180347 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:46:58.180492 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:46:58.180855 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":154,"threads_started":1}
I20260928 02:46:58.183274 6789 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41507
I20260928 02:46:58.183508 6789 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:58.184024 6789 heartbeater.cc:511] Master 127.28.248.190:41507 requested a full tablet report, sending...
I20260928 02:46:58.185216 6470 ts_manager.cc:194] Registered new tserver with Master: 81f03d5cef884d2e8b9502cecf0e8a83 (127.28.248.130:37587)
I20260928 02:46:58.185812 6470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.130:52799
W20260928 02:46:58.285986 6792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:58.286229 6792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:58.286291 6792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:58.290268 6792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:58.290320 6792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:58.290400 6792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:58.295280 6792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:58.296325 6792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:58.297758 6792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:58.307231 6797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.307217 6800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.307283 6792 server_base.cc:1034] running on GCE node
W20260928 02:46:58.307217 6798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.307827 6792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:58.308430 6792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:58.309615 6792 hybrid_clock.cc:648] HybridClock initialized: now 1790563618309590 us; error 38 us; skew 500 ppm
I20260928 02:46:58.311455 6792 webserver.cc:460] Webserver started at http://127.28.248.131:46797/ using document root <none> and password file <none>
I20260928 02:46:58.312067 6792 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:58.312157 6792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:58.312372 6792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:58.314406 6792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "aac5a2e5782144d5b09221ad0aa18851"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "543c41a2fe4ac4030a2865677e6ebe46"
server_key_iv: "f65279f48758e3a66ef552b08a70852e"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.314949 6792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "aac5a2e5782144d5b09221ad0aa18851"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "543c41a2fe4ac4030a2865677e6ebe46"
server_key_iv: "f65279f48758e3a66ef552b08a70852e"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.318567 6792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:46:58.321108 6806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.322331 6792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:58.322456 6792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "aac5a2e5782144d5b09221ad0aa18851"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "543c41a2fe4ac4030a2865677e6ebe46"
server_key_iv: "f65279f48758e3a66ef552b08a70852e"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.322566 6792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:58.352536 6792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:58.353328 6792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:58.353484 6792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:58.354033 6792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:58.354907 6792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:58.354954 6792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.354992 6792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:58.355011 6792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.363649 6792 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:35771
I20260928 02:46:58.363673 6919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:35771 every 8 connection(s)
I20260928 02:46:58.364547 6792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:58.370572 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6792
I20260928 02:46:58.370699 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:58.370956 29666 external_mini_cluster.cc:1428] Setting key 7e166b88d460ee2920024f4d5444946c
I20260928 02:46:58.374702 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.366594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:50571 (local address 127.28.248.190:41507)
0928 02:46:58.366754 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:58.366759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:58.367948 (+ 1189us) server_negotiation.cc:407] Connection header received
0928 02:46:58.369384 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:58.369387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:58.369443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:58.369531 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:58.370451 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.371235 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.372714 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.373042 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.373163 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:58.374201 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:58.374205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:58.374208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:58.374220 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:58.374249 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:58.374257 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:58.374336 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:58.374499 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:46:58.374563 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:46:58.375571 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.367117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41507 (local address 127.28.248.131:50571)
0928 02:46:58.367780 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:58.367822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:58.369157 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:58.369692 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:58.369699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:58.369783 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:58.370295 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:58.370308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.371389 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.371396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:58.372533 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:58.372543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:58.373200 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:58.373204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:58.373377 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:58.373939 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:58.373940 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:58.373983 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:58.374051 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:58.374339 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:58.374350 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:58.374357 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:46:58.374490 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:46:58.374807 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260928 02:46:58.376663 6920 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41507
I20260928 02:46:58.376917 6920 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:58.377446 6920 heartbeater.cc:511] Master 127.28.248.190:41507 requested a full tablet report, sending...
I20260928 02:46:58.378461 6470 ts_manager.cc:194] Registered new tserver with Master: aac5a2e5782144d5b09221ad0aa18851 (127.28.248.131:35771)
I20260928 02:46:58.378985 6470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.131:50571
I20260928 02:46:58.385035 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:46:58.388088 6509 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:58.387040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.28.248.190:41507)
0928 02:46:58.387214 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:58.387219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:58.387354 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:46:58.387515 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:58.387518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:58.387578 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:58.387674 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:58.387963 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50218: BlockingRecv error: recv got EOF from 127.0.0.1:50218 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:46:58.388351 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6530
I20260928 02:46:58.394044 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6661
I20260928 02:46:58.399428 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6792
I20260928 02:46:58.404665 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6439
2026-09-28T02:46:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (893 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:58Z Disabled control of system clock
I20260928 02:46:58.433861 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45283
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:46:58.545140 6931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:58.545423 6931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:58.545461 6931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:58.549458 6931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:58.549512 6931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:58.549528 6931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:58.549544 6931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:58.549559 6931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:58.554239 6931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:45283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:58.555269 6931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:58.556591 6931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:58.565019 6936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.565075 6937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.565074 6939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.565282 6931 server_base.cc:1034] running on GCE node
I20260928 02:46:58.565862 6931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:58.566946 6931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:58.568116 6931 hybrid_clock.cc:648] HybridClock initialized: now 1790563618568100 us; error 33 us; skew 500 ppm
I20260928 02:46:58.570066 6931 webserver.cc:460] Webserver started at http://127.28.248.190:33203/ using document root <none> and password file <none>
I20260928 02:46:58.570628 6931 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:58.570689 6931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:58.570853 6931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:58.572940 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "06f19990fe27a579df148c849c0a995a"
server_key_iv: "16402542e78cf3123bab8384fbcc1475"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.573455 6931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "06f19990fe27a579df148c849c0a995a"
server_key_iv: "16402542e78cf3123bab8384fbcc1475"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.577004 6931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:58.579340 6945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.580530 6931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:46:58.580691 6931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "06f19990fe27a579df148c849c0a995a"
server_key_iv: "16402542e78cf3123bab8384fbcc1475"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.580808 6931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:58.585270 6931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:58.585891 6931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:58.586072 6931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:58.592545 6931 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:45283
I20260928 02:46:58.592574 6997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:45283 every 8 connection(s)
I20260928 02:46:58.593528 6931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:58.596683 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:58.598927 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 6931
I20260928 02:46:58.599057 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:46:58.599323 29666 external_mini_cluster.cc:1428] Setting key 2cdbb3bad40d8f53f53ea6aeb620b370
I20260928 02:46:58.602417 6998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0: Bootstrap starting.
I20260928 02:46:58.604815 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.600749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.28.248.190:45283)
0928 02:46:58.601350 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:58.601364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:58.601398 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:58.602007 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:58.602023 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:58.602373 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:58.602696 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:58.603514 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:58.603518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:58.603530 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:58.603540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:58.603594 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:58.603609 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:58.603717 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:58.603818 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:46:58.604095 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260928 02:46:58.605022 6998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:58.605870 6998 log.cc:824] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:58.607786 6998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0: No bootstrap required, opened a new log
I20260928 02:46:58.611243 6998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45283 } }
I20260928 02:46:58.611471 6998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:58.611558 6998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2ef2c6a2ea4291ae5107bf897da5f0, State: Initialized, Role: FOLLOWER
I20260928 02:46:58.611985 6998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45283 } }
I20260928 02:46:58.612136 6998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:58.612200 6998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:58.612300 6998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:58.613430 6998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45283 } }
I20260928 02:46:58.613780 6998 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2ef2c6a2ea4291ae5107bf897da5f0; no voters:
I20260928 02:46:58.614140 6998 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:58.614246 7003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:58.614491 7003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [term 1 LEADER]: Becoming Leader. State: Replica: 1a2ef2c6a2ea4291ae5107bf897da5f0, State: Running, Role: LEADER
I20260928 02:46:58.614882 7003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45283 } }
I20260928 02:46:58.615371 6998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:58.616416 7004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45283 } } }
I20260928 02:46:58.616539 7004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:58.616945 7005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2ef2c6a2ea4291ae5107bf897da5f0. Latest consensus state: current_term: 1 leader_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2ef2c6a2ea4291ae5107bf897da5f0" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45283 } } }
I20260928 02:46:58.616959 7012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:58.619927 7012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:58.617100 7005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:58.625379 7012 catalog_manager.cc:1347] Generated new cluster ID: e3d44324f73549b1b1e2bbec67d762f1
I20260928 02:46:58.625439 7012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:58.636991 7012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:58.637287 7012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:58.643278 7012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2ef2c6a2ea4291ae5107bf897da5f0: Generated new TSK 0
I20260928 02:46:58.643921 7012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:58.650204 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45283
--builtin_ntp_servers=127.28.248.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:46:58.767609 7022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:58.767863 7022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:58.767908 7022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:58.771989 7022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:58.772044 7022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:58.772120 7022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:46:58.776830 7022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:58.777793 7022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:58.779067 7022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:58.788513 7027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.788514 7030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.788535 7028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.788666 7022 server_base.cc:1034] running on GCE node
I20260928 02:46:58.789201 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:58.789746 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:58.790935 7022 hybrid_clock.cc:648] HybridClock initialized: now 1790563618790926 us; error 37 us; skew 500 ppm
I20260928 02:46:58.792753 7022 webserver.cc:460] Webserver started at http://127.28.248.129:38159/ using document root <none> and password file <none>
I20260928 02:46:58.793318 7022 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:58.793406 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:58.793617 7022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:58.795578 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "daf2c8bd6c5e4b62ae05c13f654c877f"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "2403a353f3137dde335f69afc5ee17c4"
server_key_iv: "1f35d68425765e8c98c60179446065e4"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.796145 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "daf2c8bd6c5e4b62ae05c13f654c877f"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "2403a353f3137dde335f69afc5ee17c4"
server_key_iv: "1f35d68425765e8c98c60179446065e4"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.799713 7022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 02:46:58.802158 7036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.803316 7022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:58.803457 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "daf2c8bd6c5e4b62ae05c13f654c877f"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "2403a353f3137dde335f69afc5ee17c4"
server_key_iv: "1f35d68425765e8c98c60179446065e4"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.803568 7022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:58.807986 7022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:58.808564 7022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:58.808792 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:58.809312 7022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:58.810173 7022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:58.810243 7022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.810309 7022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:58.810362 7022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.818842 7022 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:45107
I20260928 02:46:58.818897 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:45107 every 8 connection(s)
I20260928 02:46:58.819746 7022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:58.821750 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7022
I20260928 02:46:58.821848 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:58.822160 29666 external_mini_cluster.cc:1428] Setting key 0e298979d93957f419754385efc43dee
I20260928 02:46:58.824304 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45283
--builtin_ntp_servers=127.28.248.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:46:58.826380 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.821523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:52145 (local address 127.28.248.190:45283)
0928 02:46:58.821679 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:58.821685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:58.822887 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 02:46:58.824482 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:58.824487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:58.824530 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:58.824678 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:58.825845 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:58.825851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:58.825857 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:58.825873 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:58.825900 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:58.825913 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:58.826015 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:58.826170 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:46:58.826218 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:46:58.827255 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.821998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45283 (local address 127.28.248.129:52145)
0928 02:46:58.822719 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:58.822769 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:58.824246 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:58.824891 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:58.824898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:58.824982 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:58.825587 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:58.825589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:58.825627 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:58.825692 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:58.826014 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:58.826022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:58.826027 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:46:58.826170 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 02:46:58.826347 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":125,"threads_started":1}
I20260928 02:46:58.828490 7150 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45283
I20260928 02:46:58.828759 7150 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:58.829242 7150 heartbeater.cc:511] Master 127.28.248.190:45283 requested a full tablet report, sending...
I20260928 02:46:58.830787 6961 ts_manager.cc:194] Registered new tserver with Master: daf2c8bd6c5e4b62ae05c13f654c877f (127.28.248.129:45107)
W20260928 02:46:58.938022 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:58.938287 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:58.938364 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:58.942351 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:58.942400 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:58.942512 7153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:46:58.947118 7153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:58.948167 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:58.949553 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:58.958868 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:58.958961 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.959065 7153 server_base.cc:1034] running on GCE node
W20260928 02:46:58.958988 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:58.959461 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:58.960053 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:58.961216 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790563618961200 us; error 27 us; skew 500 ppm
I20260928 02:46:58.963050 7153 webserver.cc:460] Webserver started at http://127.28.248.130:42283/ using document root <none> and password file <none>
I20260928 02:46:58.963629 7153 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:58.963691 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:58.963924 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:58.966030 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "dece12433ea74436b993128a2200cf8d"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "1275381d721e9d028da4dc6324ffbd6c"
server_key_iv: "e7c5b798732fc5bf7c932b08f641a73d"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.966691 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "dece12433ea74436b993128a2200cf8d"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "1275381d721e9d028da4dc6324ffbd6c"
server_key_iv: "e7c5b798732fc5bf7c932b08f641a73d"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.971274 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 02:46:58.973809 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.975013 7153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:58.975152 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "dece12433ea74436b993128a2200cf8d"
format_stamp: "Formatted at 2026-09-28 02:46:58 on dist-test-slave-r238"
server_key: "1275381d721e9d028da4dc6324ffbd6c"
server_key_iv: "e7c5b798732fc5bf7c932b08f641a73d"
server_key_version: "encryptionkey@0"
I20260928 02:46:58.975265 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:58.979750 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:58.980330 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:58.980507 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:58.981102 7153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:58.981972 7153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:58.982044 7153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.982110 7153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:58.982146 7153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:58.990924 7153 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:35263
I20260928 02:46:58.990968 7280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:35263 every 8 connection(s)
I20260928 02:46:58.991837 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:58.997664 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.993672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:56677 (local address 127.28.248.190:45283)
0928 02:46:58.993810 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:58.993813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:58.994845 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 02:46:58.996062 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:58.996064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:58.996101 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:58.996175 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:58.997286 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:58.997289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:58.997291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:58.997301 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:58.997318 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:58.997325 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:58.997399 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:58.997514 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:46:58.997560 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:46:58.998502 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:58.994070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45283 (local address 127.28.248.130:56677)
0928 02:46:58.994688 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:58.994727 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:58.995875 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:58.996328 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:58.996335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:58.996421 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:58.997040 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:58.997042 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:58.997080 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:58.997160 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:58.997398 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:58.997404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:58.997408 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:58.997535 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:46:58.997728 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260928 02:46:58.999573 7281 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45283
I20260928 02:46:58.999797 7281 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:59.000191 7281 heartbeater.cc:511] Master 127.28.248.190:45283 requested a full tablet report, sending...
I20260928 02:46:59.001295 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7153
I20260928 02:46:59.001369 6961 ts_manager.cc:194] Registered new tserver with Master: dece12433ea74436b993128a2200cf8d (127.28.248.130:35263)
I20260928 02:46:59.001420 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:59.001717 29666 external_mini_cluster.cc:1428] Setting key 385f12375834b728a78ef6490ed59746
I20260928 02:46:59.003633 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45283
--builtin_ntp_servers=127.28.248.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:46:59.117573 7284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:59.117836 7284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:59.117937 7284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:59.121934 7284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:59.122014 7284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:59.122097 7284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:46:59.126855 7284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:46:59.127939 7284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:59.129287 7284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:59.138865 7290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:59.138962 7289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:59.138986 7284 server_base.cc:1034] running on GCE node
W20260928 02:46:59.139053 7292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:59.139686 7284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:59.140256 7284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:59.141404 7284 hybrid_clock.cc:648] HybridClock initialized: now 1790563619141390 us; error 24 us; skew 500 ppm
I20260928 02:46:59.143280 7284 webserver.cc:460] Webserver started at http://127.28.248.131:38921/ using document root <none> and password file <none>
I20260928 02:46:59.143883 7284 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:59.143946 7284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:59.144111 7284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:59.146154 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "d39be95890ac4e3eaa38ad61f6ff1755"
format_stamp: "Formatted at 2026-09-28 02:46:59 on dist-test-slave-r238"
server_key: "e91693d8881f4d5f13ebcd076d2fbf38"
server_key_iv: "51053a1e8cc823b3a34babe54fe0c688"
server_key_version: "encryptionkey@0"
I20260928 02:46:59.146634 7284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "d39be95890ac4e3eaa38ad61f6ff1755"
format_stamp: "Formatted at 2026-09-28 02:46:59 on dist-test-slave-r238"
server_key: "e91693d8881f4d5f13ebcd076d2fbf38"
server_key_iv: "51053a1e8cc823b3a34babe54fe0c688"
server_key_version: "encryptionkey@0"
I20260928 02:46:59.150192 7284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:46:59.152561 7298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:59.153812 7284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:46:59.153914 7284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "d39be95890ac4e3eaa38ad61f6ff1755"
format_stamp: "Formatted at 2026-09-28 02:46:59 on dist-test-slave-r238"
server_key: "e91693d8881f4d5f13ebcd076d2fbf38"
server_key_iv: "51053a1e8cc823b3a34babe54fe0c688"
server_key_version: "encryptionkey@0"
I20260928 02:46:59.154001 7284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:59.158716 7284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:59.159281 7284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:59.159415 7284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:59.159941 7284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:59.160856 7284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:59.160904 7284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:59.160941 7284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:59.160962 7284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:59.169912 7284 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:39883
I20260928 02:46:59.169941 7411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:39883 every 8 connection(s)
I20260928 02:46:59.170928 7284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:59.177107 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:59.172838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:55003 (local address 127.28.248.190:45283)
0928 02:46:59.172984 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:59.172988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:59.173993 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 02:46:59.175318 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:59.175321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:59.175367 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:59.175470 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:59.176590 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:59.176597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:59.176599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:46:59.176637 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:59.176660 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:46:59.176670 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:59.176775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:59.176920 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:46:59.176965 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:46:59.177909 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:59.173196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45283 (local address 127.28.248.131:55003)
0928 02:46:59.173841 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:59.173881 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:59.175096 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:59.175642 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:59.175649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:59.175736 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:59.176338 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:46:59.176339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:59.176379 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:46:59.176446 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:59.176768 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:59.176776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:59.176783 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:46:59.176964 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:46:59.177194 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260928 02:46:59.179029 7412 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45283
I20260928 02:46:59.179273 7412 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:59.179719 7412 heartbeater.cc:511] Master 127.28.248.190:45283 requested a full tablet report, sending...
I20260928 02:46:59.180044 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7284
I20260928 02:46:59.180127 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:59.180452 29666 external_mini_cluster.cc:1428] Setting key c33cb9f2a235677539c1e72d47059512
I20260928 02:46:59.180876 6961 ts_manager.cc:194] Registered new tserver with Master: d39be95890ac4e3eaa38ad61f6ff1755 (127.28.248.131:39883)
I20260928 02:46:59.182521 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:46:59.185261 7001 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:59.184370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.28.248.190:45283)
0928 02:46:59.184477 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:59.184481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:59.184703 (+ 222us) server_negotiation.cc:407] Connection header received
0928 02:46:59.184814 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:59.184816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:59.184844 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:59.184907 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:59.185147 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42410: BlockingRecv error: recv got EOF from 127.0.0.1:42410 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:46:59.185582 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7022
I20260928 02:46:59.191501 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7153
I20260928 02:46:59.196733 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7284
I20260928 02:46:59.201957 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 6931
2026-09-28T02:46:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (796 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:59 dist-test-slave-r238 krb5kdc[7422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:59 dist-test-slave-r238 krb5kdc[7422](info): set up 2 sockets
Sep 28 02:46:59 dist-test-slave-r238 krb5kdc[7422](info): commencing operation
krb5kdc: starting...
W20260928 02:47:00.242086 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:47:00 dist-test-slave-r238 krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:47:00.368103 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43123
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:43123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:00.480816 7435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:00.481163 7435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:00.481240 7435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:00.485499 7435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:00.485599 7435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:00.485633 7435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:00.485675 7435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:00.485711 7435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:00.491303 7435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:43123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:00.492473 7435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:00.493636 7435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:00.502499 7440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:00.502506 7441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:00.502619 7435 server_base.cc:1034] running on GCE node
W20260928 02:47:00.502518 7443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:00.503257 7435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:00.504464 7435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:00.505663 7435 hybrid_clock.cc:648] HybridClock initialized: now 1790563620505640 us; error 34 us; skew 500 ppm
Sep 28 02:47:00 dist-test-slave-r238 krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563620, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:00.508483 7435 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:47:00.509402 7435 webserver.cc:460] Webserver started at http://127.28.248.190:34135/ using document root <none> and password file <none>
I20260928 02:47:00.509967 7435 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:00.510058 7435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:00.510295 7435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:00.512326 7435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "5eac58ce99e04cde9a9f516679544b37"
format_stamp: "Formatted at 2026-09-28 02:47:00 on dist-test-slave-r238"
server_key: "788d11fee4be0f5734c1d47c87997b4b"
server_key_iv: "eb44db137ae5b00cc36d693b62b28a9b"
server_key_version: "encryptionkey@0"
I20260928 02:47:00.512912 7435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "5eac58ce99e04cde9a9f516679544b37"
format_stamp: "Formatted at 2026-09-28 02:47:00 on dist-test-slave-r238"
server_key: "788d11fee4be0f5734c1d47c87997b4b"
server_key_iv: "eb44db137ae5b00cc36d693b62b28a9b"
server_key_version: "encryptionkey@0"
I20260928 02:47:00.516592 7435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:00.519106 7450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:00.520337 7435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:47:00.520480 7435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "5eac58ce99e04cde9a9f516679544b37"
format_stamp: "Formatted at 2026-09-28 02:47:00 on dist-test-slave-r238"
server_key: "788d11fee4be0f5734c1d47c87997b4b"
server_key_iv: "eb44db137ae5b00cc36d693b62b28a9b"
server_key_version: "encryptionkey@0"
I20260928 02:47:00.520629 7435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:00.545499 7435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:00.548990 7435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:00.549204 7435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:00.556677 7435 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:43123
I20260928 02:47:00.556735 7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:43123 every 8 connection(s)
I20260928 02:47:00.557755 7435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:00.560738 7503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:00.565013 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7435
I20260928 02:47:00.565142 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:47:00.565407 29666 external_mini_cluster.cc:1428] Setting key 52a73bd4ce94257d1eebfe56adb35161
I20260928 02:47:00.566605 7503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: Bootstrap starting.
I20260928 02:47:00.569011 7503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:00.569847 7503 log.cc:824] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:00.571974 7503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: No bootstrap required, opened a new log
Sep 28 02:47:00 dist-test-slave-r238 krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:00.576893 7503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eac58ce99e04cde9a9f516679544b37" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43123 } }
I20260928 02:47:00.577250 7503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:00.577306 7503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eac58ce99e04cde9a9f516679544b37, State: Initialized, Role: FOLLOWER
I20260928 02:47:00.577786 7503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eac58ce99e04cde9a9f516679544b37" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43123 } }
I20260928 02:47:00.577955 7503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:00.578025 7503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:00.578148 7503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:00.579386 7503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eac58ce99e04cde9a9f516679544b37" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43123 } }
I20260928 02:47:00.579784 7503 leader_election.cc:302] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eac58ce99e04cde9a9f516679544b37; no voters:
I20260928 02:47:00.580127 7503 leader_election.cc:288] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:00.580314 7508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:00.580647 7508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [term 1 LEADER]: Becoming Leader. State: Replica: 5eac58ce99e04cde9a9f516679544b37, State: Running, Role: LEADER
I20260928 02:47:00.580727 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:00.566843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.28.248.190:43123)
0928 02:47:00.567262 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:00.567280 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:00.567326 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:47:00.567969 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:00.567984 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:00.568328 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:00.568668 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:00.569544 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:00.570579 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:00.571702 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:00.572215 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:00.572349 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:00.575123 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:00.575132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:00.575157 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:00.575198 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:00.577346 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:00.577829 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:00.577835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:00.577840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:00.577984 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:00.578318 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:00.578323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:00.578324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:00.578867 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:00.579067 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:00.579338 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:47:00.579664 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":147,"threads_started":1}
I20260928 02:47:00.581066 7508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eac58ce99e04cde9a9f516679544b37" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43123 } }
I20260928 02:47:00.581451 7503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:00.582625 7509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eac58ce99e04cde9a9f516679544b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eac58ce99e04cde9a9f516679544b37" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43123 } } }
I20260928 02:47:00.582825 7509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:00.583128 7509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eac58ce99e04cde9a9f516679544b37. Latest consensus state: current_term: 1 leader_uuid: "5eac58ce99e04cde9a9f516679544b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eac58ce99e04cde9a9f516679544b37" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43123 } } }
I20260928 02:47:00.583261 7509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37 [sys.catalog]: This master's current role is: LEADER
W20260928 02:47:00.586854 7523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:47:00.586928 7523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:47:00.587006 7519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:00.587710 7519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:00.593422 7519 catalog_manager.cc:1347] Generated new cluster ID: d094cf480917438aa6b112a481268bc3
I20260928 02:47:00.593494 7519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:00.620795 7519 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:00.621902 7519 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:00.629696 7519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: Generated new TSK 0
I20260928 02:47:00.630443 7519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:47:00.664870 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43123
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:00.800789 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:00.801170 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:00.801234 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:00.805378 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:00.805451 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:00.805512 7529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:47:00.810799 7529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:00.811944 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:00.812971 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:00.823271 7537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:00.823333 7534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:00.823340 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:00.823441 7529 server_base.cc:1034] running on GCE node
I20260928 02:47:00.824059 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:00.824859 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:00.826053 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790563620826029 us; error 45 us; skew 500 ppm
Sep 28 02:47:00 dist-test-slave-r238 krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563620, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:00.829062 7529 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:47:00.830044 7529 webserver.cc:460] Webserver started at http://127.28.248.129:41679/ using document root <none> and password file <none>
I20260928 02:47:00.830664 7529 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:00.830762 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:00.831002 7529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:00.833253 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "89cb7f72a2414c53ae20dfbc3a58d108"
format_stamp: "Formatted at 2026-09-28 02:47:00 on dist-test-slave-r238"
server_key: "3858227cc4ac870c702229a2ff21f802"
server_key_iv: "24213d7c8fee15700068aae5ae922357"
server_key_version: "encryptionkey@0"
I20260928 02:47:00.833842 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "89cb7f72a2414c53ae20dfbc3a58d108"
format_stamp: "Formatted at 2026-09-28 02:47:00 on dist-test-slave-r238"
server_key: "3858227cc4ac870c702229a2ff21f802"
server_key_iv: "24213d7c8fee15700068aae5ae922357"
server_key_version: "encryptionkey@0"
I20260928 02:47:00.837822 7529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:47:00.840310 7544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:00.841567 7529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:00.841709 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "89cb7f72a2414c53ae20dfbc3a58d108"
format_stamp: "Formatted at 2026-09-28 02:47:00 on dist-test-slave-r238"
server_key: "3858227cc4ac870c702229a2ff21f802"
server_key_iv: "24213d7c8fee15700068aae5ae922357"
server_key_version: "encryptionkey@0"
I20260928 02:47:00.841826 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:00.863003 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:00.866519 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:00.866746 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:00.867372 7529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:00.868275 7529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:00.868348 7529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:00.868422 7529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:00.868474 7529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:00.877391 7529 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:33421
I20260928 02:47:00.877408 7657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:33421 every 8 connection(s)
I20260928 02:47:00.878315 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:00.883002 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7529
I20260928 02:47:00.883136 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:00.883419 29666 external_mini_cluster.cc:1428] Setting key 12720856ee86ad265a080388d50bd228
Sep 28 02:47:00 dist-test-slave-r238 krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563620, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:47:00.893895 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:00.880158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:50381 (local address 127.28.248.190:43123)
0928 02:47:00.880309 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:00.880313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:00.881167 (+ 854us) server_negotiation.cc:407] Connection header received
0928 02:47:00.882718 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:00.882755 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:00.882823 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:00.882904 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:00.884450 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:00.885223 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:00.886351 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:00.886692 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:00.886778 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:00.889805 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:00.889812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:00.889819 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:00.889853 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:00.891615 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:00.892140 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:00.892146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:00.892149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:00.892248 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:00.892594 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:00.892598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:00.892599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:00.892870 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:00.892987 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:00.893527 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 02:47:00.893667 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:47:00.894680 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:00.880477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43123 (local address 127.28.248.129:50381)
0928 02:47:00.880984 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:00.881027 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:00.882504 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:00.883124 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:00.883133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:00.883643 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:00.884290 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:00.884310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:00.885345 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:00.885347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:00.886227 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:00.886236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:00.886851 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:00.886858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:00.887090 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:00.887743 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:00.887769 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:00.889629 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:00.891744 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:00.891749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:00.891762 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:00.892025 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:00.892355 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:00.892357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:00.892359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:00.892495 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:00.892979 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:00.892985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:00.893270 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 02:47:00.893438 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:47:00.893712 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":179,"threads_started":1}
I20260928 02:47:00.896008 7658 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43123
I20260928 02:47:00.896231 7658 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:47:00.896837 7658 heartbeater.cc:511] Master 127.28.248.190:43123 requested a full tablet report, sending...
I20260928 02:47:00.898537 7467 ts_manager.cc:194] Registered new tserver with Master: 89cb7f72a2414c53ae20dfbc3a58d108 (127.28.248.129:33421)
I20260928 02:47:00.899868 7467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:50381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:47:00.915650 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43123
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:01.032588 7663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:01.032897 7663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:01.032941 7663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:01.036995 7663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:01.037062 7663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:01.037124 7663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:47:01.042575 7663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:01.043617 7663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:01.044566 7663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:01.054658 7669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:01.054621 7668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:01.054721 7663 server_base.cc:1034] running on GCE node
W20260928 02:47:01.054605 7671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:01.055249 7663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:01.055897 7663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:01.057085 7663 hybrid_clock.cc:648] HybridClock initialized: now 1790563621057050 us; error 45 us; skew 500 ppm
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:01.059949 7663 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:47:01.060906 7663 webserver.cc:460] Webserver started at http://127.28.248.130:33491/ using document root <none> and password file <none>
I20260928 02:47:01.061575 7663 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:01.061636 7663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:01.061865 7663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:01.063992 7663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "1d79821227864ae3819513722a662355"
format_stamp: "Formatted at 2026-09-28 02:47:01 on dist-test-slave-r238"
server_key: "43e9cfa489b50c13625fd20542b8aef6"
server_key_iv: "6738059d6ea4ed94f8ed8e41915c8d55"
server_key_version: "encryptionkey@0"
I20260928 02:47:01.064669 7663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "1d79821227864ae3819513722a662355"
format_stamp: "Formatted at 2026-09-28 02:47:01 on dist-test-slave-r238"
server_key: "43e9cfa489b50c13625fd20542b8aef6"
server_key_iv: "6738059d6ea4ed94f8ed8e41915c8d55"
server_key_version: "encryptionkey@0"
I20260928 02:47:01.068825 7663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:47:01.071471 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:01.072937 7663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:01.073122 7663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "1d79821227864ae3819513722a662355"
format_stamp: "Formatted at 2026-09-28 02:47:01 on dist-test-slave-r238"
server_key: "43e9cfa489b50c13625fd20542b8aef6"
server_key_iv: "6738059d6ea4ed94f8ed8e41915c8d55"
server_key_version: "encryptionkey@0"
I20260928 02:47:01.073241 7663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:01.096632 7663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:01.100035 7663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:01.100263 7663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:01.100907 7663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:01.101775 7663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:01.101819 7663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:01.101893 7663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:01.101933 7663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:01.110801 7663 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:38679
I20260928 02:47:01.110822 7791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:38679 every 8 connection(s)
I20260928 02:47:01.111773 7663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:01.112495 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7663
I20260928 02:47:01.112659 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:01.113029 29666 external_mini_cluster.cc:1428] Setting key 69c3e58ea39f26394875f82f689284dc
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:01.132033 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.113859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:57023 (local address 127.28.248.190:43123)
0928 02:47:01.113998 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.114003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.115108 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 02:47:01.117052 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.117055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.117109 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.117190 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:01.118824 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.119496 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.121410 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.121636 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.121724 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.127780 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:01.127787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:01.127794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:01.127833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:01.129728 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:01.130361 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:01.130364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:01.130365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:01.130442 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:01.130820 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:01.130823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:01.130824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:01.131025 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:01.131140 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.131729 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.131865 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:47:01.133137 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.114405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43123 (local address 127.28.248.130:57023)
0928 02:47:01.114929 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.114990 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.116832 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.117372 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.117379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.117920 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:01.118640 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.118664 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.119640 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.119645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.121267 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.121280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.121823 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.121830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.122054 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.122761 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:01.122812 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:01.127558 (+ 4746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:01.129866 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:01.129872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:01.129886 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:01.130233 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:01.130536 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:01.130542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:01.130544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:01.130714 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:01.131138 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:01.131145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:01.131451 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.131668 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.132030 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260928 02:47:01.134384 7792 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43123
I20260928 02:47:01.134684 7792 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:01.135205 7792 heartbeater.cc:511] Master 127.28.248.190:43123 requested a full tablet report, sending...
I20260928 02:47:01.136324 7467 ts_manager.cc:194] Registered new tserver with Master: 1d79821227864ae3819513722a662355 (127.28.248.130:38679)
I20260928 02:47:01.137044 7467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:57023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:47:01.144881 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43123
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:01.261152 7797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:01.261446 7797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:01.261492 7797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:01.265611 7797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:01.265682 7797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:01.265777 7797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:47:01.271473 7797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:01.272686 7797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:01.273734 7797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:01.284552 7805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:01.284552 7802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:01.284766 7797 server_base.cc:1034] running on GCE node
W20260928 02:47:01.284556 7803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:01.285418 7797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:01.286166 7797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:01.287401 7797 hybrid_clock.cc:648] HybridClock initialized: now 1790563621287378 us; error 36 us; skew 500 ppm
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:01.296952 7797 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:47:01.298137 7797 webserver.cc:460] Webserver started at http://127.28.248.131:40505/ using document root <none> and password file <none>
I20260928 02:47:01.298790 7797 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:01.298893 7797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:01.299165 7797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:01.301559 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e"
format_stamp: "Formatted at 2026-09-28 02:47:01 on dist-test-slave-r238"
server_key: "b844f6878901a2eb92e84372871195dd"
server_key_iv: "a7e96473feca5680dcb25f24fa4462ea"
server_key_version: "encryptionkey@0"
I20260928 02:47:01.302307 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e"
format_stamp: "Formatted at 2026-09-28 02:47:01 on dist-test-slave-r238"
server_key: "b844f6878901a2eb92e84372871195dd"
server_key_iv: "a7e96473feca5680dcb25f24fa4462ea"
server_key_version: "encryptionkey@0"
I20260928 02:47:01.306938 7797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:47:01.309903 7812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:01.311347 7797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:01.311507 7797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e"
format_stamp: "Formatted at 2026-09-28 02:47:01 on dist-test-slave-r238"
server_key: "b844f6878901a2eb92e84372871195dd"
server_key_iv: "a7e96473feca5680dcb25f24fa4462ea"
server_key_version: "encryptionkey@0"
I20260928 02:47:01.311648 7797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:01.334360 7797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:01.338402 7797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:01.338650 7797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:01.339327 7797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:01.340425 7797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:01.340503 7797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:01.340610 7797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:01.340729 7797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:01.351143 7797 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:44247
I20260928 02:47:01.351161 7925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:44247 every 8 connection(s)
I20260928 02:47:01.352165 7797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:01.362592 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 7797
I20260928 02:47:01.362735 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563621, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:01.363135 29666 external_mini_cluster.cc:1428] Setting key 926edcada32b88c1b8c26958ad3bbff7
I20260928 02:47:01.367949 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.354308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:56157 (local address 127.28.248.190:43123)
0928 02:47:01.354466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.354470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.355409 (+ 939us) server_negotiation.cc:407] Connection header received
0928 02:47:01.356999 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.357003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.357076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.357158 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:01.358492 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.359048 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.360306 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.360542 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.360690 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.363857 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:01.363861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:01.363864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:01.363899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:01.365793 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:01.366339 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:01.366342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:01.366343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:01.366424 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:01.366823 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:01.366826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:01.366827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:01.367076 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:01.367167 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.367711 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.367827 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:47:01.369004 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.354740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:43123 (local address 127.28.248.131:56157)
0928 02:47:01.355245 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.355293 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.356759 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.357350 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.357358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.357773 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:01.358316 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.358331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.359205 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.359208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.360164 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.360173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.360755 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.360759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.360949 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.361644 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:01.361663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:01.363627 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:01.365923 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:01.365929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:01.365945 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:01.366248 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:01.366506 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:01.366508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:01.366509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:01.366710 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:01.367172 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:01.367178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:01.367445 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.367632 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.367924 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":144,"threads_started":1}
I20260928 02:47:01.370292 7926 heartbeater.cc:347] Connected to a master server at 127.28.248.190:43123
I20260928 02:47:01.370530 7926 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:01.371054 7926 heartbeater.cc:511] Master 127.28.248.190:43123 requested a full tablet report, sending...
I20260928 02:47:01.372208 7467 ts_manager.cc:194] Registered new tserver with Master: d5d1be3f7bef4bd794cb4e40f5f5290e (127.28.248.131:44247)
I20260928 02:47:01.372959 7467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:56157
I20260928 02:47:01.377949 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:01.389057 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.380038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.28.248.190:43123)
0928 02:47:01.380196 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.380200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.380435 (+ 235us) server_negotiation.cc:407] Connection header received
0928 02:47:01.380528 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.380530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.380574 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.380746 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:01.381539 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.382472 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.383435 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.383709 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.383828 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.385336 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:01.385339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:01.385341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:01.385366 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:01.387027 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:01.387542 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:01.387545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:01.387546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:01.387634 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:01.387926 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:01.387929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:01.387930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:01.388256 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:01.388372 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.388769 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.388881 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:47:01.392472 7467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:01.395784 7467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:01.411223 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.404956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.28.248.129:33421)
0928 02:47:01.405624 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.405632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.405955 (+ 323us) server_negotiation.cc:407] Connection header received
0928 02:47:01.406042 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.406050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.406265 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.406402 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:01.406745 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.408045 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.409692 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.410481 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.410613 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.410775 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.410950 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.411059 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":107,"threads_started":1}
I20260928 02:47:01.411262 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.404712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38679 (local address 127.0.0.1:33808)
0928 02:47:01.405406 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.405461 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.405598 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.406424 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.406431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.406457 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:01.406613 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.406621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.408203 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.408206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.409816 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.409827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.410891 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.410897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.410979 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.411001 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.411061 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.411112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":215,"thread_start_us":83,"threads_started":1}
I20260928 02:47:01.411314 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.404788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.28.248.130:38679)
0928 02:47:01.405450 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.405462 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.405743 (+ 281us) server_negotiation.cc:407] Connection header received
0928 02:47:01.405830 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.405835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.406069 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.406223 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:01.406745 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.408070 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.409951 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.410764 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.410911 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.411036 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.411082 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.411155 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":109,"threads_started":1}
I20260928 02:47:01.411262 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.404777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:33421 (local address 127.0.0.1:55294)
0928 02:47:01.405744 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.405758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.405855 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.406479 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.406481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.406494 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:01.406617 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.406621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.408201 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.408206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.409563 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.409572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.410616 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.410632 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.410720 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.410764 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.410902 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.411091 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":145,"thread_start_us":117,"threads_started":1}
I20260928 02:47:01.413093 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.405452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:44247 (local address 127.0.0.1:33290)
0928 02:47:01.405757 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.405770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.405841 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.406729 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.406732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.406749 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:01.406841 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.406850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.408170 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.408176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.409817 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.409827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.412658 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.412662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.412759 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.412784 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.412865 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.412942 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":100,"thread_start_us":85,"threads_started":1}
I20260928 02:47:01.413173 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.405903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.28.248.131:44247)
0928 02:47:01.406304 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.406319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.406340 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:47:01.406412 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.406416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.406590 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.406736 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:01.406967 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.408010 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.411511 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.412345 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.412463 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.412568 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.412922 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.413025 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":134,"threads_started":1}
I20260928 02:47:01.414444 7592 tablet_service.cc:1467] Processing CreateTablet for tablet f1b07c4e7d744fac92f4029833fc16d2 (DEFAULT_TABLE table=test-table [id=f448d0a0e37c4beca7ee730691ed975f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:01.414448 7726 tablet_service.cc:1467] Processing CreateTablet for tablet f1b07c4e7d744fac92f4029833fc16d2 (DEFAULT_TABLE table=test-table [id=f448d0a0e37c4beca7ee730691ed975f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:01.415477 7726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b07c4e7d744fac92f4029833fc16d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:01.415518 7592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b07c4e7d744fac92f4029833fc16d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:01.417836 7860 tablet_service.cc:1467] Processing CreateTablet for tablet f1b07c4e7d744fac92f4029833fc16d2 (DEFAULT_TABLE table=test-table [id=f448d0a0e37c4beca7ee730691ed975f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:01.418919 7860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b07c4e7d744fac92f4029833fc16d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:01.422353 7943 tablet_bootstrap.cc:492] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: Bootstrap starting.
I20260928 02:47:01.424782 7943 tablet_bootstrap.cc:654] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:01.425626 7943 log.cc:824] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:01.425799 7944 tablet_bootstrap.cc:492] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Bootstrap starting.
I20260928 02:47:01.426586 7945 tablet_bootstrap.cc:492] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Bootstrap starting.
I20260928 02:47:01.428138 7943 tablet_bootstrap.cc:492] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: No bootstrap required, opened a new log
I20260928 02:47:01.428367 7943 ts_tablet_manager.cc:1397] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260928 02:47:01.428692 7944 tablet_bootstrap.cc:654] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:01.428756 7945 tablet_bootstrap.cc:654] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:01.429668 7944 log.cc:824] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:01.430033 7945 log.cc:824] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:01.431641 7944 tablet_bootstrap.cc:492] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: No bootstrap required, opened a new log
I20260928 02:47:01.431855 7944 ts_tablet_manager.cc:1397] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260928 02:47:01.432053 7945 tablet_bootstrap.cc:492] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: No bootstrap required, opened a new log
I20260928 02:47:01.432235 7945 ts_tablet_manager.cc:1397] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 02:47:01.432817 7943 raft_consensus.cc:348] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.433115 7943 raft_consensus.cc:374] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:01.433205 7943 raft_consensus.cc:729] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cb7f72a2414c53ae20dfbc3a58d108, State: Initialized, Role: FOLLOWER
I20260928 02:47:01.433813 7943 consensus_queue.cc:260] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.434553 7658 heartbeater.cc:503] Master 127.28.248.190:43123 was elected leader, sending a full tablet report...
I20260928 02:47:01.434907 7943 ts_tablet_manager.cc:1428] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 02:47:01.435588 7944 raft_consensus.cc:348] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.435873 7944 raft_consensus.cc:374] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:01.435984 7944 raft_consensus.cc:729] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d79821227864ae3819513722a662355, State: Initialized, Role: FOLLOWER
I20260928 02:47:01.436411 7944 consensus_queue.cc:260] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.437096 7792 heartbeater.cc:503] Master 127.28.248.190:43123 was elected leader, sending a full tablet report...
I20260928 02:47:01.437223 7945 raft_consensus.cc:348] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.437424 7944 ts_tablet_manager.cc:1428] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260928 02:47:01.437448 7945 raft_consensus.cc:374] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:01.437505 7945 raft_consensus.cc:729] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d1be3f7bef4bd794cb4e40f5f5290e, State: Initialized, Role: FOLLOWER
I20260928 02:47:01.437954 7945 consensus_queue.cc:260] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.438702 7926 heartbeater.cc:503] Master 127.28.248.190:43123 was elected leader, sending a full tablet report...
I20260928 02:47:01.438902 7945 ts_tablet_manager.cc:1428] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:47:01.581759 7949 raft_consensus.cc:482] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:01.582034 7949 raft_consensus.cc:504] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.583421 7949 leader_election.cc:288] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d79821227864ae3819513722a662355 (127.28.248.130:38679), d5d1be3f7bef4bd794cb4e40f5f5290e (127.28.248.131:44247)
I20260928 02:47:01.590161 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.584737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:52123 (local address 127.28.248.130:38679)
0928 02:47:01.584893 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.584898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.584912 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:47:01.584996 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.585001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.585046 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.585142 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:01.585638 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.586494 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.588349 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.589312 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.589468 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.589537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.589910 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.590018 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:47:01.590193 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.584094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:43319 (local address 127.28.248.131:44247)
0928 02:47:01.584223 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.584228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.584515 (+ 287us) server_negotiation.cc:407] Connection header received
0928 02:47:01.584653 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.584656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.584714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.584811 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:01.585952 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.586824 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.588682 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.589407 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.589579 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.589634 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.590020 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.590089 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:47:01.590392 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.583662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38679 (local address 127.28.248.129:52123)
0928 02:47:01.584147 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.584164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.584302 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.585190 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.585195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.585215 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:01.585458 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.585466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.586635 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.586641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.588108 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.588120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.589616 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.589622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.589750 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.589786 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.589878 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.589971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":15,"thread_start_us":108,"threads_started":1}
I20260928 02:47:01.590760 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.583691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:44247 (local address 127.28.248.129:43319)
0928 02:47:01.584427 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.584443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:01.584533 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:01.585249 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:01.585251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:01.585276 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:01.585458 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.585466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.586948 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.586951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.588108 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:01.588120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.589724 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.589728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:01.589877 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.589902 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:47:01.590337 (+ 435us) client_negotiation.cc:240] Negotiation successful
0928 02:47:01.590439 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":113,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260928 02:47:01.591625 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b07c4e7d744fac92f4029833fc16d2" candidate_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d79821227864ae3819513722a662355" is_pre_election: true
I20260928 02:47:01.591643 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b07c4e7d744fac92f4029833fc16d2" candidate_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" is_pre_election: true
I20260928 02:47:01.592020 7746 raft_consensus.cc:2454] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cb7f72a2414c53ae20dfbc3a58d108 in term 0.
I20260928 02:47:01.592036 7880 raft_consensus.cc:2454] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cb7f72a2414c53ae20dfbc3a58d108 in term 0.
I20260928 02:47:01.592561 7546 leader_election.cc:302] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d79821227864ae3819513722a662355, 89cb7f72a2414c53ae20dfbc3a58d108; no voters:
I20260928 02:47:01.592942 7949 raft_consensus.cc:2781] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:01.593043 7949 raft_consensus.cc:482] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:01.593108 7949 raft_consensus.cc:3037] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:01.594616 7949 raft_consensus.cc:504] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.595042 7949 leader_election.cc:288] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [CANDIDATE]: Term 1 election: Requested vote from peers 1d79821227864ae3819513722a662355 (127.28.248.130:38679), d5d1be3f7bef4bd794cb4e40f5f5290e (127.28.248.131:44247)
I20260928 02:47:01.595397 7746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b07c4e7d744fac92f4029833fc16d2" candidate_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d79821227864ae3819513722a662355"
I20260928 02:47:01.595443 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b07c4e7d744fac92f4029833fc16d2" candidate_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e"
I20260928 02:47:01.595577 7880 raft_consensus.cc:3037] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:01.595639 7746 raft_consensus.cc:3037] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:01.597012 7880 raft_consensus.cc:2454] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cb7f72a2414c53ae20dfbc3a58d108 in term 1.
I20260928 02:47:01.597347 7547 leader_election.cc:302] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cb7f72a2414c53ae20dfbc3a58d108, d5d1be3f7bef4bd794cb4e40f5f5290e; no voters:
I20260928 02:47:01.597419 7746 raft_consensus.cc:2454] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cb7f72a2414c53ae20dfbc3a58d108 in term 1.
I20260928 02:47:01.597637 7949 raft_consensus.cc:2781] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:01.598004 7949 raft_consensus.cc:686] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [term 1 LEADER]: Becoming Leader. State: Replica: 89cb7f72a2414c53ae20dfbc3a58d108, State: Running, Role: LEADER
I20260928 02:47:01.598385 7949 consensus_queue.cc:237] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } }
I20260928 02:47:01.602279 7465 catalog_manager.cc:5495] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89cb7f72a2414c53ae20dfbc3a58d108 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cb7f72a2414c53ae20dfbc3a58d108" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 } health_report { overall_health: UNKNOWN } } }
W20260928 02:47:01.603163 7927 tablet.cc:2369] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:47:01.613042 7793 tablet.cc:2369] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:47:01.629756 7659 tablet.cc:2369] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:01.648335 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.644326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.28.248.129:33421)
0928 02:47:01.644448 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:01.644452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:01.644467 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:47:01.644681 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:01.644684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:01.644772 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:01.644898 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:01.645204 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.645724 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.646745 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:01.646975 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:01.647080 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:01.647437 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:01.647914 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:01.648059 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:01.648119 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:47:01.648182 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:47:01.654922 7746 raft_consensus.cc:1261] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 1 FOLLOWER]: Refusing update from remote peer 89cb7f72a2414c53ae20dfbc3a58d108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:01.654922 7880 raft_consensus.cc:1261] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 1 FOLLOWER]: Refusing update from remote peer 89cb7f72a2414c53ae20dfbc3a58d108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:01.655668 7954 consensus_queue.cc:1035] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5d1be3f7bef4bd794cb4e40f5f5290e" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:01.655862 7949 consensus_queue.cc:1035] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d79821227864ae3819513722a662355" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:01.663997 7959 mvcc.cc:205] Tried to move back new op lower bound from 7334148594290626560 to 7334148594069778432. Current Snapshot: MvccSnapshot[applied={T|T < 7334148594290626560}]
I20260928 02:47:01.665359 7958 mvcc.cc:205] Tried to move back new op lower bound from 7334148594290626560 to 7334148594069778432. Current Snapshot: MvccSnapshot[applied={T|T < 7334148594290626560}]
I20260928 02:47:01.672688 7466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:01.672866 7466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:01.675442 7466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5eac58ce99e04cde9a9f516679544b37: Sending DeleteTablet for 3 replicas of tablet f1b07c4e7d744fac92f4029833fc16d2
I20260928 02:47:01.676429 7726 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b07c4e7d744fac92f4029833fc16d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:01 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:33808
I20260928 02:47:01.676680 7592 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b07c4e7d744fac92f4029833fc16d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:01 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:55294
I20260928 02:47:01.676680 7860 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b07c4e7d744fac92f4029833fc16d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:01 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:33290
I20260928 02:47:01.677182 7967 tablet_replica.cc:330] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: stopping tablet replica
I20260928 02:47:01.677399 7969 tablet_replica.cc:330] T f1b07c4e7d744fac92f4029833fc16d2 P 89cb7f72a2414c53ae20dfbc3a58d108: stopping tablet replica
I20260928 02:47:01.677692 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7529
I20260928 02:47:01.679116 7968 tablet_replica.cc:330] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: stopping tablet replica
I20260928 02:47:01.679454 7968 raft_consensus.cc:2229] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:01.679823 7968 raft_consensus.cc:2258] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:01.677584 7967 raft_consensus.cc:2229] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:01.680392 7967 raft_consensus.cc:2258] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:01.681671 7968 ts_tablet_manager.cc:1905] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:01.681793 7967 ts_tablet_manager.cc:1905] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:47:01.683888 7453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:33421 (error 108)
I20260928 02:47:01.684600 7967 ts_tablet_manager.cc:1918] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:01.684600 7968 ts_tablet_manager.cc:1918] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:01.684733 7967 log.cc:1196] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/f1b07c4e7d744fac92f4029833fc16d2
I20260928 02:47:01.684733 7968 log.cc:1196] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/f1b07c4e7d744fac92f4029833fc16d2
I20260928 02:47:01.685024 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.684684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:33421 (local address 127.0.0.1:55318)
0928 02:47:01.684867 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.684936 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:47:01.685065 7967 ts_tablet_manager.cc:1939] T f1b07c4e7d744fac92f4029833fc16d2 P 1d79821227864ae3819513722a662355: Deleting consensus metadata
I20260928 02:47:01.685065 7968 ts_tablet_manager.cc:1939] T f1b07c4e7d744fac92f4029833fc16d2 P d5d1be3f7bef4bd794cb4e40f5f5290e: Deleting consensus metadata
W20260928 02:47:01.685221 7453 catalog_manager.cc:4584] TS 89cb7f72a2414c53ae20dfbc3a58d108 (127.28.248.129:33421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1b07c4e7d744fac92f4029833fc16d2: Network error: Client connection negotiation failed: client connection to 127.28.248.129:33421: connect: Connection refused (error 111)
I20260928 02:47:01.685739 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7663
I20260928 02:47:01.688131 7453 catalog_manager.cc:4841] TS d5d1be3f7bef4bd794cb4e40f5f5290e (127.28.248.131:44247): tablet f1b07c4e7d744fac92f4029833fc16d2 (table test-table [id=f448d0a0e37c4beca7ee730691ed975f]) successfully deleted
I20260928 02:47:01.691445 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7797
I20260928 02:47:01.693221 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:01.691604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38679 (local address 127.0.0.1:33812)
0928 02:47:01.693022 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:01.693107 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.130:38679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1229}
I20260928 02:47:01.697530 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 7435
2026-09-28T02:47:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2496 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7973](info): set up 2 sockets
Sep 28 02:47:01 dist-test-slave-r238 krb5kdc[7973](info): commencing operation
krb5kdc: starting...
W20260928 02:47:02.741925 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:47:02 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/rangeradmin_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/rangeradmin_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/rangerlookup_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/rangerlookup_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
I20260928 02:47:02.914774 29666 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9u0ndH/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/postgres -l logfile start
2026-09-28 02:47:05.634 UTC [8003] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:47:05.634 UTC [8003] LOG: listening on IPv4 address "127.28.248.148", port 55851
2026-09-28 02:47:05.639 UTC [8003] LOG: listening on Unix socket "/tmp/.s.PGSQL.55851"
2026-09-28 02:47:05.674 UTC [8006] LOG: database system was shut down at 2026-09-28 02:47:04 UTC
2026-09-28 02:47:05.678 UTC [8003] LOG: database system is ready to accept connections
I20260928 02:47:06.625572 29666 mini_postgres.cc:96] Postgres bound to 55851
2026-09-28 02:47:06.631 UTC [8014] FATAL: database "slave" does not exist
127.28.248.148:55851 - accepting connections
I20260928 02:47:06.632632 29666 mini_ranger.cc:161] Starting Ranger...
I20260928 02:47:06.647619 29666 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:47:07.142252 29666 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:47:07.142361 29666 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-admin
2026-09-28 02:47:07,437 [I] DB FLAVOR :POSTGRES
2026-09-28 02:47:07,437 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:47:07,437 [I] Checking connection..
2026-09-28 02:47:07,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:47:07,702 [I] Checking connection passed.
2026-09-28 02:47:07,702 [I] --------- Verifying version history table ---------
2026-09-28 02:47:07,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:47:07,934 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:47:07,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:47:08,188 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:47:08,188 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:47:08,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:47:08.437 UTC [8078] WARNING: there is no transaction in progress
2026-09-28 02:47:08,443 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:47:08,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:47:08,700 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:47:08,700 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:47:08,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:47:08,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:47:09,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r238.c.gcp-upstream.internal','N') ;"
2026-09-28 02:47:09,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:47:09,679 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:47:09,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:47:09,935 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:47:09,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:47:10,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:47:10,454 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:47:10,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:47:11.500 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.517 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.535 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.552 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.570 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.590 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.661 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.670 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.689 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.700 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.711 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.725 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.735 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.742 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.750 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:11.757 UTC [8186] WARNING: there is no transaction in progress
2026-09-28 02:47:12,480 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:47:12,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:47:12,721 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:47:12,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:47:12,974 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:47:12,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:47:13,244 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:47:13,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:47:13,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:47:13,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r238.c.gcp-upstream.internal';"
2026-09-28 02:47:14,004 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:47:14,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:47:14,267 [I] DB_PATCHES have already been applied
I20260928 02:47:14.273694 29666 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-admin:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9u0ndH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:47:14.273769 29666 mini_ranger.cc:193] Using host: 127.28.248.148
I20260928 02:47:14.277434 29666 mini_ranger.cc:227] Ranger admin URL: http://127.28.248.148:55899
Sep 28, 2026 2:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:47:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:47:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:47:15.412998 29666 mini_ranger.cc:160] Time spent starting Ranger: real 8.780s user 0.000s sys 0.010s
Sep 28, 2026 2:47:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:47:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 2:47:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:47:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:47:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:47:37.464893 29666 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/rangerkms_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/rangerkms_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.248.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.248.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:47:37 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563657, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:47:37.567871 29666 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:47:37.578935 29666 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:47:38.442972 29666 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:47:38.443092 29666 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms
I20260928 02:47:38.443223 29666 mini_ranger_kms.cc:227] Using postgres at 127.28.248.148:55851
2026-09-28 02:47:38,624 [I] DB FLAVOR :POSTGRES
2026-09-28 02:47:38,624 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:47:38,624 [I] Checking connection
2026-09-28 02:47:38,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:47:38,859 [I] connection success
2026-09-28 02:47:38,859 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:47:38,859 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:47:38,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:47:39,096 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:47:39,096 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:47:39,096 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:47:39,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:55851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:47:39,344 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:47:39.775846 29666 mini_ranger_kms.cc:310] Created kms service
I20260928 02:47:39.986707 29666 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:47:40.030337 29666 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:47:40 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563657, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:47:40.433256 29666 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:47:40.433416 29666 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms:/tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:47:40.433444 29666 mini_ranger_kms.cc:240] Using host: 127.28.248.148
I20260928 02:47:40.437016 29666 mini_ranger_kms.cc:276] Ranger KMS PID: 8366
I20260928 02:47:40.437161 29666 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.248.148:40015
Sep 28, 2026 2:47:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:47:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:47:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:47:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:47:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:47:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:47:42.717562 29666 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.150s user 0.013s sys 0.151s
I20260928 02:47:42.717724 29666 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:47:42.717842 29666 mini_ranger_kms.cc:401] 127.28.248.148:40015/kms/v1/keys
Sep 28 02:47:42 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563662, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.248.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:47:43 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563662, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.248.148@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
Sep 28 02:47:43 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563662, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.248.148@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
Sep 28 02:47:45 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:47:45.732666 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41103
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:41103
--ranger_config_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:40015/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:45.853070 8417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:45.853374 8417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:45.853444 8417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:45.857897 8417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:45.857986 8417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:45.858008 8417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:45.858032 8417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:45.858049 8417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:45.863941 8417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.248.148:40015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:41103
--ranger_config_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:45.865156 8417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:45.866214 8417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:45.875494 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:45.875593 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:45.875505 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:45.875881 8417 server_base.cc:1034] running on GCE node
I20260928 02:47:45.876559 8417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:45.877985 8417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:45.879185 8417 hybrid_clock.cc:648] HybridClock initialized: now 1790563665879171 us; error 45 us; skew 500 ppm
Sep 28 02:47:45 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:45.882463 8417 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:47:45.883641 8417 webserver.cc:460] Webserver started at http://127.28.248.190:43069/ using document root <none> and password file <none>
I20260928 02:47:45.884275 8417 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:45.884337 8417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:45.884596 8417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:47:45 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:47:45.953569 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "465b1775a3ee493aa62ddc9dceb3301e"
format_stamp: "Formatted at 2026-09-28 02:47:45 on dist-test-slave-r238"
server_key: "fbdbba24658bd37aa0c7b30f3dd1a009"
server_key_iv: "a3b9d82013bb4795d67e83a2748c9da6"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:45.954276 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "465b1775a3ee493aa62ddc9dceb3301e"
format_stamp: "Formatted at 2026-09-28 02:47:45 on dist-test-slave-r238"
server_key: "fbdbba24658bd37aa0c7b30f3dd1a009"
server_key_iv: "a3b9d82013bb4795d67e83a2748c9da6"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:45.958879 8417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:47:45.977350 8432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:45.978853 8417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:45.978979 8417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "465b1775a3ee493aa62ddc9dceb3301e"
format_stamp: "Formatted at 2026-09-28 02:47:45 on dist-test-slave-r238"
server_key: "fbdbba24658bd37aa0c7b30f3dd1a009"
server_key_iv: "a3b9d82013bb4795d67e83a2748c9da6"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:45.979144 8417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:45.992894 8417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:45.996202 8417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:45.996414 8417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:46.005566 8417 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:41103
I20260928 02:47:46.005582 8484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:41103 every 8 connection(s)
I20260928 02:47:46.006706 8417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:46.010902 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:46.011384 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 8417
I20260928 02:47:46.011531 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:47:46.011850 29666 external_mini_cluster.cc:1428] Setting key d1f1900e4fa1f9508aed992517fb8a23
I20260928 02:47:46.020685 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e: Bootstrap starting.
Sep 28 02:47:46 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:46.024483 8485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:46.026180 8485 log.cc:824] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:46.029016 8485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e: No bootstrap required, opened a new log
I20260928 02:47:46.029417 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:46.013662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.28.248.190:41103)
0928 02:47:46.014191 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:46.014261 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:46.014307 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:47:46.015319 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:46.015343 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:46.015732 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:46.016162 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:46.017339 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:46.018557 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:46.019732 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:46.020283 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:46.020497 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:46.023659 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:46.023666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:46.023681 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:46.023718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:46.025824 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:46.026519 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:46.026526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:46.026532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:46.026705 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:46.027118 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:46.027122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:46.027124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:46.027589 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:46.027826 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:46.028174 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 02:47:46.028504 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":178,"threads_started":1}
I20260928 02:47:46.033566 8485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465b1775a3ee493aa62ddc9dceb3301e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41103 } }
I20260928 02:47:46.033882 8485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:46.033937 8485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465b1775a3ee493aa62ddc9dceb3301e, State: Initialized, Role: FOLLOWER
I20260928 02:47:46.034499 8485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465b1775a3ee493aa62ddc9dceb3301e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41103 } }
I20260928 02:47:46.034669 8485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:46.034747 8485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:46.034852 8485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:46.036141 8485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465b1775a3ee493aa62ddc9dceb3301e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41103 } }
I20260928 02:47:46.036530 8485 leader_election.cc:302] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 465b1775a3ee493aa62ddc9dceb3301e; no voters:
I20260928 02:47:46.036935 8485 leader_election.cc:288] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:46.037088 8490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:46.037348 8490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [term 1 LEADER]: Becoming Leader. State: Replica: 465b1775a3ee493aa62ddc9dceb3301e, State: Running, Role: LEADER
I20260928 02:47:46.037747 8490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465b1775a3ee493aa62ddc9dceb3301e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41103 } }
I20260928 02:47:46.038323 8485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:46.040421 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 465b1775a3ee493aa62ddc9dceb3301e. Latest consensus state: current_term: 1 leader_uuid: "465b1775a3ee493aa62ddc9dceb3301e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465b1775a3ee493aa62ddc9dceb3301e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41103 } } }
I20260928 02:47:46.040553 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:46.040760 8485 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:47:46.040786 8491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "465b1775a3ee493aa62ddc9dceb3301e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "465b1775a3ee493aa62ddc9dceb3301e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41103 } } }
I20260928 02:47:46.040879 8491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:46.042277 8504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:46.049170 8504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:46.058600 8504 catalog_manager.cc:1347] Generated new cluster ID: fce11cd915414af7a2ddfc64d8fa1f3f
I20260928 02:47:46.058857 8504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:46.073580 8504 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:46.074985 8504 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:46.083829 8504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e: Generated new TSK 0
I20260928 02:47:46.084798 8504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:47:46.829495 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:40015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41103
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:46.972239 8530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:46.972657 8530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:46.972730 8530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:46.978838 8530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:46.978926 8530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:46.979017 8530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:47:46.987108 8530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.248.148:40015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:46.988579 8530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:46.989681 8530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:47.000347 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:47.000520 8530 server_base.cc:1034] running on GCE node
W20260928 02:47:47.000660 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:47.000655 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:47.001327 8530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:47.002022 8530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:47.003228 8530 hybrid_clock.cc:648] HybridClock initialized: now 1790563667003189 us; error 48 us; skew 500 ppm
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:47.013078 8530 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:47:47.014243 8530 webserver.cc:460] Webserver started at http://127.28.248.129:43039/ using document root <none> and password file <none>
I20260928 02:47:47.014971 8530 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:47.015065 8530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:47.015295 8530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:47:47.043238 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "0f18ec509e5241f7af4e6636f40d25a2"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "20aa55a1547b21e39c057ef072279ced"
server_key_iv: "081a997380ca2e35805243d67aff6518"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.044121 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "0f18ec509e5241f7af4e6636f40d25a2"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "20aa55a1547b21e39c057ef072279ced"
server_key_iv: "081a997380ca2e35805243d67aff6518"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.048652 8530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:47:47.072593 8549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.074677 8530 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260928 02:47:47.074885 8530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "0f18ec509e5241f7af4e6636f40d25a2"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "20aa55a1547b21e39c057ef072279ced"
server_key_iv: "081a997380ca2e35805243d67aff6518"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.075110 8530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:47.098905 8530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:47.103037 8530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:47.103302 8530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:47.104034 8530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:47.105381 8530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:47.105484 8530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.105604 8530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:47.105697 8530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.120779 8530 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:43005
I20260928 02:47:47.122203 8530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:47.127593 8662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:43005 every 8 connection(s)
I20260928 02:47:47.129696 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 8530
I20260928 02:47:47.129843 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:47.130106 29666 external_mini_cluster.cc:1428] Setting key 0a807f8b7e510bc9b62f54da580db6c7
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:47.157414 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.128500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41103 (local address 127.28.248.129:46911)
0928 02:47:47.137167 (+ 8667us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:47.137237 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:47.138555 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:47.141484 (+ 2929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:47.141495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:47.141870 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:47.142456 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.142471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.144729 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.144733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.146293 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.146304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.149089 (+ 2785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.149093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.149315 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.150034 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:47.150056 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:47.151817 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:47.154147 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:47.154154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:47.154184 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:47.154514 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:47.154817 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:47.154820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:47.154822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:47.154990 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:47.155464 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:47.155471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:47.155775 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 02:47:47.155965 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:47:47.156302 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8394,"thread_start_us":143,"threads_started":1}
I20260928 02:47:47.158159 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.128299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:46911 (local address 127.28.248.190:41103)
0928 02:47:47.136832 (+ 8533us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.136839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.138907 (+ 2068us) server_negotiation.cc:407] Connection header received
0928 02:47:47.138962 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.138967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.139022 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.139111 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:47.142620 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.143477 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.146439 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.146817 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.146971 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.152018 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:47.152022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:47.152025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:47.152052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:47.154016 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.154628 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.154631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.154633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.154721 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.155092 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.155095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.155097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.155319 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:47.155411 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.157867 (+ 2456us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.157995 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8427,"thread_start_us":64,"threads_started":1}
I20260928 02:47:47.159953 8663 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41103
I20260928 02:47:47.160207 8663 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:47.160792 8663 heartbeater.cc:511] Master 127.28.248.190:41103 requested a full tablet report, sending...
I20260928 02:47:47.162513 8444 ts_manager.cc:194] Registered new tserver with Master: 0f18ec509e5241f7af4e6636f40d25a2 (127.28.248.129:43005)
I20260928 02:47:47.163982 8444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:46911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:47:47.184490 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:40015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41103
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:47.335467 8669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:47.335824 8669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:47.335888 8669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:47.342156 8669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:47.342243 8669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:47.342330 8669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:47:47.350178 8669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.248.148:40015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:47.351611 8669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:47.352962 8669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:47.366581 8678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:47.369787 8676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:47.369829 8675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:47.370198 8669 server_base.cc:1034] running on GCE node
I20260928 02:47:47.370692 8669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:47.371534 8669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:47.372691 8669 hybrid_clock.cc:648] HybridClock initialized: now 1790563667372671 us; error 45 us; skew 500 ppm
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:47.376267 8669 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:47:47.377354 8669 webserver.cc:460] Webserver started at http://127.28.248.130:45479/ using document root <none> and password file <none>
I20260928 02:47:47.378047 8669 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:47.378134 8669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:47.378378 8669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:47:47.396597 8669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "6460a1a35bbb4be987ae71cf6d5b0b90"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "7884822c6268ad5028a685a46b9a9a17"
server_key_iv: "3305d8621c0a35e36451155230627391"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.397337 8669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "6460a1a35bbb4be987ae71cf6d5b0b90"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "7884822c6268ad5028a685a46b9a9a17"
server_key_iv: "3305d8621c0a35e36451155230627391"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.415840 8669 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.005s sys 0.000s
I20260928 02:47:47.440593 8685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.442341 8669 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 02:47:47.442481 8669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "6460a1a35bbb4be987ae71cf6d5b0b90"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "7884822c6268ad5028a685a46b9a9a17"
server_key_iv: "3305d8621c0a35e36451155230627391"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.442615 8669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:47.464047 8669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:47.467943 8669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:47.468222 8669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:47.468971 8669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:47.470101 8669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:47.470212 8669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.470316 8669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:47.470387 8669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.484675 8669 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:33915
I20260928 02:47:47.485904 8669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:47.487406 8798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:33915 every 8 connection(s)
I20260928 02:47:47.494995 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 8669
I20260928 02:47:47.495132 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:47.495440 29666 external_mini_cluster.cc:1428] Setting key 52aea8064842877a028caf8e41b0b03d
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:47.512511 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.490777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:60231 (local address 127.28.248.190:41103)
0928 02:47:47.490930 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.490934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.494692 (+ 3758us) server_negotiation.cc:407] Connection header received
0928 02:47:47.500658 (+ 5966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.500662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.500722 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.500809 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:47.502422 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.503229 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.505038 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.505386 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.505459 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.508476 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:47.508480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:47.508483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:47.508508 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:47.510213 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.510827 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.510830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.510832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.510911 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.511289 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.511292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.511294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.511494 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:47.511558 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.512214 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.512343 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:47:47.514369 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.491240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41103 (local address 127.28.248.130:60231)
0928 02:47:47.494478 (+ 3238us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:47.494540 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:47.500313 (+ 5773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:47.501156 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:47.501167 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:47.501533 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:47.502242 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.502262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.503394 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.503397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.504902 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.504913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.505667 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.505670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.505861 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.506569 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:47.506589 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:47.508292 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:47.510337 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:47.510343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:47.510359 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:47.510695 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:47.511017 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:47.511020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:47.511022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:47.511183 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:47.511648 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:47.511654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:47.511950 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 02:47:47.513012 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0928 02:47:47.513350 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2996,"thread_start_us":140,"threads_started":1}
I20260928 02:47:47.515872 8799 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41103
I20260928 02:47:47.516106 8799 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:47.516673 8799 heartbeater.cc:511] Master 127.28.248.190:41103 requested a full tablet report, sending...
I20260928 02:47:47.517894 8444 ts_manager.cc:194] Registered new tserver with Master: 6460a1a35bbb4be987ae71cf6d5b0b90 (127.28.248.130:33915)
I20260928 02:47:47.518620 8444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:60231
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:47:47.550941 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:40015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41103
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
W20260928 02:47:47.715015 8804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:47.715315 8804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:47.715392 8804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:47.719538 8804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:47.719620 8804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:47.719722 8804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:47:47.725175 8804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.248.148:40015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:41103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:47:47.726275 8804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:47.727306 8804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:47.737739 8812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:47.737938 8809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:47.738188 8810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:47.738435 8804 server_base.cc:1034] running on GCE node
I20260928 02:47:47.738945 8804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:47.739691 8804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:47.740890 8804 hybrid_clock.cc:648] HybridClock initialized: now 1790563667740863 us; error 38 us; skew 500 ppm
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:47.744671 8804 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:47:47.745813 8804 webserver.cc:460] Webserver started at http://127.28.248.131:36627/ using document root <none> and password file <none>
I20260928 02:47:47.746474 8804 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:47.746553 8804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:47.746784 8804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:47:47.761511 8804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "952342b9b19e42ce9b49917b6515b2b4"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "d5b26a90ed84bbfa9d43f78a784ecc1f"
server_key_iv: "6616c4a8ec68fb82871fc4768090327d"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.762315 8804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "952342b9b19e42ce9b49917b6515b2b4"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "d5b26a90ed84bbfa9d43f78a784ecc1f"
server_key_iv: "6616c4a8ec68fb82871fc4768090327d"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.769078 8804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 02:47:47.792313 8819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.794133 8804 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:47:47.794270 8804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "952342b9b19e42ce9b49917b6515b2b4"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-r238"
server_key: "d5b26a90ed84bbfa9d43f78a784ecc1f"
server_key_iv: "6616c4a8ec68fb82871fc4768090327d"
server_key_version: "kuduclusterkey@0"
I20260928 02:47:47.794407 8804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:47.807482 8804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:47.810912 8804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:47.811085 8804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:47.811771 8804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:47.812878 8804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:47.812939 8804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.812992 8804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:47.813022 8804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.824254 8804 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:39311
I20260928 02:47:47.825551 8804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:47.828886 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 8804
I20260928 02:47:47.829010 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:47.829317 29666 external_mini_cluster.cc:1428] Setting key ff9840bac7ae91d0b769dda05264e635
I20260928 02:47:47.826331 8932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:39311 every 8 connection(s)
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:47:47.845129 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.828267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:39195 (local address 127.28.248.190:41103)
0928 02:47:47.829724 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.829729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.832113 (+ 2384us) server_negotiation.cc:407] Connection header received
0928 02:47:47.833299 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.833302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.833352 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.833399 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:47.834903 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.835689 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.837280 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.837609 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.837689 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.840747 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:47.840750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:47.840753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:47.840779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:47.842651 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.843273 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.843277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.843279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.843365 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.843771 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.843774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.843776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.843985 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:47.844082 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.844758 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.844905 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1349}
I20260928 02:47:47.846019 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.828757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41103 (local address 127.28.248.131:39195)
0928 02:47:47.831980 (+ 3223us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:47.832029 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:47.833073 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:47.833793 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:47.833802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:47.834116 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:47.834754 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.834772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.835832 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.835835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.837144 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.837155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.837919 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.837928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.838116 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.838783 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:47.838802 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:47.840570 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:47.842798 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:47.842803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:47.842820 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:47.843148 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:47.843476 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:47.843479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:47.843481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:47.843659 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:47.844127 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:47.844133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:47.844425 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:47:47.844658 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 02:47:47.844996 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3022,"thread_start_us":125,"threads_started":1}
I20260928 02:47:47.847303 8933 heartbeater.cc:347] Connected to a master server at 127.28.248.190:41103
I20260928 02:47:47.847558 8933 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:47.848094 8933 heartbeater.cc:511] Master 127.28.248.190:41103 requested a full tablet report, sending...
I20260928 02:47:47.849246 8444 ts_manager.cc:194] Registered new tserver with Master: 952342b9b19e42ce9b49917b6515b2b4 (127.28.248.131:39311)
I20260928 02:47:47.849913 8444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:39195
I20260928 02:47:47.854465 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:47.866820 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.857336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.28.248.190:41103)
0928 02:47:47.857442 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.857446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.857673 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:47:47.857848 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.857851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.857896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.857946 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:47.858864 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.859601 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.860768 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.861069 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.861142 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.863078 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:47.863082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:47.863085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:47.863110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:47.864991 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.865482 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.865485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.865487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.865572 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:47.865905 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:47.865909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:47.865911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:47.866107 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:47.866205 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.866542 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.866651 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:47:47.870491 8444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:47.875001 8444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 02:47:47 dist-test-slave-r238 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:47:47.907400 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.886447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:39311 (local address 127.0.0.1:45470)
0928 02:47:47.889206 (+ 2759us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:47.889217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:47.889279 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:47.897643 (+ 8364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:47.897646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:47.897664 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:47.897763 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.897768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.904007 (+ 6239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.904013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.905557 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.905567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.906859 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.906862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.906986 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.907028 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:47:47.907129 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:47:47.907245 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2698,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260928 02:47:47.908051 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.887224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.28.248.131:39311)
0928 02:47:47.890171 (+ 2947us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.890181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.890200 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:47:47.890266 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.890270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.890434 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.897492 (+ 7058us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:47.897899 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.903902 (+ 6003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.905720 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.906536 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.906643 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.906804 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.907781 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.907892 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2854,"thread_start_us":119,"threads_started":1}
I20260928 02:47:47.908452 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.885480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:43005 (local address 127.0.0.1:45206)
0928 02:47:47.888845 (+ 3365us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:47.888891 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:47.889004 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:47.896749 (+ 7745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:47.896755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:47.896782 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:47.897201 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.897213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.904422 (+ 7209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.904425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.905885 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.905893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.908077 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.908084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.908188 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.908211 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:47:47.908287 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:47:47.908353 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3192,"mutex_wait_us":298,"thread_start_us":125,"threads_started":1}
I20260928 02:47:47.911275 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.886925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.28.248.129:43005)
0928 02:47:47.889477 (+ 2552us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.889483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.889504 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:47:47.889573 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.889578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.889789 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.889920 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:47.900903 (+ 10983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.904288 (+ 3385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.906980 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.907791 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.907903 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.908012 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.911030 (+ 3018us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.911133 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2459,"thread_start_us":73,"threads_started":1}
I20260928 02:47:47.911480 8849 tablet_service.cc:1467] Processing CreateTablet for tablet 595ffd8727d94f2dbd25d97eb504d4db (DEFAULT_TABLE table=test-table [id=66f23a55af7b4238944b24dac7af6701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:47.912204 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.887605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33915 (local address 127.0.0.1:38962)
0928 02:47:47.905667 (+ 18062us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:47.905682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:47.906008 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:47.906756 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:47.906759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:47.906776 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:47.906861 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.906866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.909974 (+ 3108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.909978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.910857 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:47.910866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.911867 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.911870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:47.911947 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.911967 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:47:47.912027 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:47:47.912088 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17979,"thread_start_us":94,"threads_started":1}
I20260928 02:47:47.912271 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.887981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.28.248.130:33915)
0928 02:47:47.888330 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.888345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.905773 (+ 17428us) server_negotiation.cc:407] Connection header received
0928 02:47:47.906154 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.906160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.906435 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.906581 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:47.908886 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.909655 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.910997 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:47.911740 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:47.911881 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:47.912002 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.912055 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.912137 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260928 02:47:47.913499 8849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595ffd8727d94f2dbd25d97eb504d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:47.915342 8732 tablet_service.cc:1467] Processing CreateTablet for tablet 595ffd8727d94f2dbd25d97eb504d4db (DEFAULT_TABLE table=test-table [id=66f23a55af7b4238944b24dac7af6701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:47.916406 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595ffd8727d94f2dbd25d97eb504d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:47.921383 8952 tablet_bootstrap.cc:492] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Bootstrap starting.
I20260928 02:47:47.922778 8597 tablet_service.cc:1467] Processing CreateTablet for tablet 595ffd8727d94f2dbd25d97eb504d4db (DEFAULT_TABLE table=test-table [id=66f23a55af7b4238944b24dac7af6701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:47.923820 8597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 595ffd8727d94f2dbd25d97eb504d4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:47.924162 8952 tablet_bootstrap.cc:654] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:47.925485 8952 log.cc:824] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:47.932243 8954 tablet_bootstrap.cc:492] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2: Bootstrap starting.
I20260928 02:47:47.935861 8954 tablet_bootstrap.cc:654] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:47.937350 8954 log.cc:824] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:47.938344 8955 tablet_bootstrap.cc:492] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Bootstrap starting.
I20260928 02:47:47.938542 8952 tablet_bootstrap.cc:492] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: No bootstrap required, opened a new log
I20260928 02:47:47.938810 8952 ts_tablet_manager.cc:1397] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20260928 02:47:47.940827 8955 tablet_bootstrap.cc:654] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:47.942313 8955 log.cc:824] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:47.942938 8952 raft_consensus.cc:348] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:47.943367 8952 raft_consensus.cc:374] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:47.943470 8952 raft_consensus.cc:729] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952342b9b19e42ce9b49917b6515b2b4, State: Initialized, Role: FOLLOWER
I20260928 02:47:47.944087 8952 consensus_queue.cc:260] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:47.944901 8955 tablet_bootstrap.cc:492] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: No bootstrap required, opened a new log
I20260928 02:47:47.945102 8955 ts_tablet_manager.cc:1397] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:47:47.949194 8955 raft_consensus.cc:348] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:47.949466 8955 raft_consensus.cc:374] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:47.949519 8955 raft_consensus.cc:729] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6460a1a35bbb4be987ae71cf6d5b0b90, State: Initialized, Role: FOLLOWER
I20260928 02:47:47.949975 8955 consensus_queue.cc:260] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:47.951048 8955 ts_tablet_manager.cc:1428] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:47:47.951416 8799 heartbeater.cc:503] Master 127.28.248.190:41103 was elected leader, sending a full tablet report...
I20260928 02:47:47.952893 8954 tablet_bootstrap.cc:492] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2: No bootstrap required, opened a new log
I20260928 02:47:47.953130 8954 ts_tablet_manager.cc:1397] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260928 02:47:47.957711 8954 raft_consensus.cc:348] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:47.958042 8954 raft_consensus.cc:374] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:47.958142 8954 raft_consensus.cc:729] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f18ec509e5241f7af4e6636f40d25a2, State: Initialized, Role: FOLLOWER
I20260928 02:47:47.958616 8954 consensus_queue.cc:260] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:47.959270 8663 heartbeater.cc:503] Master 127.28.248.190:41103 was elected leader, sending a full tablet report...
I20260928 02:47:47.959609 8954 ts_tablet_manager.cc:1428] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:47:47.960434 8952 ts_tablet_manager.cc:1428] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260928 02:47:47.960850 8933 heartbeater.cc:503] Master 127.28.248.190:41103 was elected leader, sending a full tablet report...
W20260928 02:47:47.989616 8800 tablet.cc:2369] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:48.017994 8961 raft_consensus.cc:482] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:48.018250 8961 raft_consensus.cc:504] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:48.019817 8961 leader_election.cc:288] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 952342b9b19e42ce9b49917b6515b2b4 (127.28.248.131:39311), 6460a1a35bbb4be987ae71cf6d5b0b90 (127.28.248.130:33915)
I20260928 02:47:48.026337 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:48.020269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:59465 (local address 127.28.248.131:39311)
0928 02:47:48.020399 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:48.020404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:48.021750 (+ 1346us) server_negotiation.cc:407] Connection header received
0928 02:47:48.021973 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:48.021977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:48.022027 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:48.022077 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:48.022539 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.023341 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.025002 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.025611 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.025709 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:48.025764 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:48.026120 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:47:48.026211 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:47:48.026973 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:48.020209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:39311 (local address 127.28.248.129:59465)
0928 02:47:48.021656 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:48.021671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:48.021836 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:48.022244 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:48.022247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:48.022267 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:48.022410 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:48.022417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.023508 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.023511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:48.024863 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:48.024873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.025873 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.025877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:48.025970 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:48.025994 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:47:48.026737 (+ 743us) client_negotiation.cc:240] Negotiation successful
0928 02:47:48.026821 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":109,"threads_started":1}
I20260928 02:47:48.027773 8869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595ffd8727d94f2dbd25d97eb504d4db" candidate_uuid: "0f18ec509e5241f7af4e6636f40d25a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952342b9b19e42ce9b49917b6515b2b4" is_pre_election: true
I20260928 02:47:48.028105 8869 raft_consensus.cc:2454] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f18ec509e5241f7af4e6636f40d25a2 in term 0.
I20260928 02:47:48.028753 8552 leader_election.cc:302] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f18ec509e5241f7af4e6636f40d25a2, 952342b9b19e42ce9b49917b6515b2b4; no voters:
I20260928 02:47:48.029057 8961 raft_consensus.cc:2781] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:48.029134 8961 raft_consensus.cc:482] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:48.029183 8961 raft_consensus.cc:3037] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:48.030539 8961 raft_consensus.cc:504] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:48.030917 8961 leader_election.cc:288] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [CANDIDATE]: Term 1 election: Requested vote from peers 952342b9b19e42ce9b49917b6515b2b4 (127.28.248.131:39311), 6460a1a35bbb4be987ae71cf6d5b0b90 (127.28.248.130:33915)
I20260928 02:47:48.031476 8869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595ffd8727d94f2dbd25d97eb504d4db" candidate_uuid: "0f18ec509e5241f7af4e6636f40d25a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952342b9b19e42ce9b49917b6515b2b4"
I20260928 02:47:48.031594 8869 raft_consensus.cc:3037] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:48.032824 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:48.020750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:50167 (local address 127.28.248.130:33915)
0928 02:47:48.020873 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:48.020877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:48.026869 (+ 5992us) server_negotiation.cc:407] Connection header received
0928 02:47:48.027068 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:48.027073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:48.027123 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:48.027167 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:48.029371 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.030066 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.031497 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.032089 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.032196 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:48.032243 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:48.032576 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:47:48.032690 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:47:48.032943 8869 raft_consensus.cc:2454] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f18ec509e5241f7af4e6636f40d25a2 in term 1.
I20260928 02:47:48.033197 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:48.020649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33915 (local address 127.28.248.129:50167)
0928 02:47:48.026780 (+ 6131us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:48.026793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:48.026935 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:48.027308 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:48.027311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:48.027330 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:48.027437 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:48.027442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.030224 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.030227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:48.031364 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:48.031375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.032338 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.032341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:48.032433 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:48.032456 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:47:48.032956 (+ 500us) client_negotiation.cc:240] Negotiation successful
0928 02:47:48.033031 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6068,"thread_start_us":87,"threads_started":1}
I20260928 02:47:48.033586 8552 leader_election.cc:302] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f18ec509e5241f7af4e6636f40d25a2, 952342b9b19e42ce9b49917b6515b2b4; no voters:
I20260928 02:47:48.033998 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595ffd8727d94f2dbd25d97eb504d4db" candidate_uuid: "0f18ec509e5241f7af4e6636f40d25a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90"
I20260928 02:47:48.034010 8751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "595ffd8727d94f2dbd25d97eb504d4db" candidate_uuid: "0f18ec509e5241f7af4e6636f40d25a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" is_pre_election: true
I20260928 02:47:48.034252 8749 raft_consensus.cc:3037] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:48.035811 8749 raft_consensus.cc:2454] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f18ec509e5241f7af4e6636f40d25a2 in term 1.
I20260928 02:47:48.036029 8961 raft_consensus.cc:2781] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:48.036262 8961 raft_consensus.cc:686] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [term 1 LEADER]: Becoming Leader. State: Replica: 0f18ec509e5241f7af4e6636f40d25a2, State: Running, Role: LEADER
I20260928 02:47:48.036593 8961 consensus_queue.cc:237] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } }
I20260928 02:47:48.040153 8445 catalog_manager.cc:5495] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f18ec509e5241f7af4e6636f40d25a2 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "0f18ec509e5241f7af4e6636f40d25a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f18ec509e5241f7af4e6636f40d25a2" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 43005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 } health_report { overall_health: UNKNOWN } } }
I20260928 02:47:48.056731 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:48.051567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.28.248.129:43005)
0928 02:47:48.051692 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:48.051697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:48.051931 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:47:48.051992 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:48.051994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:48.052041 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:48.052150 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:48.052394 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.053213 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.054429 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:48.054770 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:48.054887 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:48.055331 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:48.055858 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:48.055989 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:48.056465 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 02:47:48.056526 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:47:48.063378 8749 raft_consensus.cc:1261] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 1 FOLLOWER]: Refusing update from remote peer 0f18ec509e5241f7af4e6636f40d25a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:48.063455 8869 raft_consensus.cc:1261] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 1 FOLLOWER]: Refusing update from remote peer 0f18ec509e5241f7af4e6636f40d25a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:48.064033 8961 consensus_queue.cc:1035] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952342b9b19e42ce9b49917b6515b2b4" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:48.064517 8961 consensus_queue.cc:1035] T 595ffd8727d94f2dbd25d97eb504d4db P 0f18ec509e5241f7af4e6636f40d25a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6460a1a35bbb4be987ae71cf6d5b0b90" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:48.075301 8968 mvcc.cc:205] Tried to move back new op lower bound from 7334148784380878848 to 7334148784281473024. Current Snapshot: MvccSnapshot[applied={T|T < 7334148784380878848}]
I20260928 02:47:48.081236 8967 mvcc.cc:205] Tried to move back new op lower bound from 7334148784380878848 to 7334148784281473024. Current Snapshot: MvccSnapshot[applied={T|T < 7334148784380878848}]
I20260928 02:47:48.081465 8970 mvcc.cc:205] Tried to move back new op lower bound from 7334148784380878848 to 7334148784281473024. Current Snapshot: MvccSnapshot[applied={T|T < 7334148784380878848}]
I20260928 02:47:48.092474 8445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:48.092698 8445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:48.095561 8445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 465b1775a3ee493aa62ddc9dceb3301e: Sending DeleteTablet for 3 replicas of tablet 595ffd8727d94f2dbd25d97eb504d4db
I20260928 02:47:48.096668 8732 tablet_service.cc:1514] Processing DeleteTablet for tablet 595ffd8727d94f2dbd25d97eb504d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:48 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:38962
I20260928 02:47:48.097282 8597 tablet_service.cc:1514] Processing DeleteTablet for tablet 595ffd8727d94f2dbd25d97eb504d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:48 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:45206
I20260928 02:47:48.097309 8849 tablet_service.cc:1514] Processing DeleteTablet for tablet 595ffd8727d94f2dbd25d97eb504d4db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:48 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:45470
I20260928 02:47:48.098541 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 8530
I20260928 02:47:48.104875 8976 tablet_replica.cc:330] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: stopping tablet replica
I20260928 02:47:48.105355 8976 raft_consensus.cc:2229] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:48.105834 8976 raft_consensus.cc:2258] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:48.107765 8976 ts_tablet_manager.cc:1905] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:48.110872 8976 ts_tablet_manager.cc:1918] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:48.111049 8976 log.cc:1196] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/595ffd8727d94f2dbd25d97eb504d4db
I20260928 02:47:48.111498 8976 ts_tablet_manager.cc:1939] T 595ffd8727d94f2dbd25d97eb504d4db P 6460a1a35bbb4be987ae71cf6d5b0b90: Deleting consensus metadata
I20260928 02:47:48.112972 8436 catalog_manager.cc:4841] TS 6460a1a35bbb4be987ae71cf6d5b0b90 (127.28.248.130:33915): tablet 595ffd8727d94f2dbd25d97eb504d4db (table test-table [id=66f23a55af7b4238944b24dac7af6701]) successfully deleted
I20260928 02:47:48.113433 8978 tablet_replica.cc:330] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: stopping tablet replica
I20260928 02:47:48.113837 8978 raft_consensus.cc:2229] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:48.114256 8978 raft_consensus.cc:2258] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:48.115909 8978 ts_tablet_manager.cc:1905] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:47:48.116672 8435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:43005 (error 108)
I20260928 02:47:48.118240 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 8669
I20260928 02:47:48.120441 8978 ts_tablet_manager.cc:1918] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:48.120638 8978 log.cc:1196] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/595ffd8727d94f2dbd25d97eb504d4db
I20260928 02:47:48.121025 8978 ts_tablet_manager.cc:1939] T 595ffd8727d94f2dbd25d97eb504d4db P 952342b9b19e42ce9b49917b6515b2b4: Deleting consensus metadata
I20260928 02:47:48.122327 8435 catalog_manager.cc:4841] TS 952342b9b19e42ce9b49917b6515b2b4 (127.28.248.131:39311): tablet 595ffd8727d94f2dbd25d97eb504d4db (table test-table [id=66f23a55af7b4238944b24dac7af6701]) successfully deleted
I20260928 02:47:48.123291 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:48.122910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:43005 (local address 127.0.0.1:45226)
0928 02:47:48.123106 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:48.123190 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260928 02:47:48.123538 8435 catalog_manager.cc:4584] TS 0f18ec509e5241f7af4e6636f40d25a2 (127.28.248.129:43005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 595ffd8727d94f2dbd25d97eb504d4db: Network error: Client connection negotiation failed: client connection to 127.28.248.129:43005: connect: Connection refused (error 111)
I20260928 02:47:48.126392 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 8804
I20260928 02:47:48.133072 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 8417
I20260928 02:47:48.138507 29666 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:47:48.473405 29666 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:47:48.473531 29666 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:47:48.818357 29666 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:47:48.818 UTC [8003] LOG: received smart shutdown request
2026-09-28 02:47:48.822 UTC [8003] LOG: background worker "logical replication launcher" (PID 8012) exited with exit code 1
2026-09-28 02:47:48.823 UTC [8007] LOG: shutting down
2026-09-28 02:47:48.998 UTC [8003] LOG: database system is shut down
2026-09-28T02:47:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47416 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:49 dist-test-slave-r238 krb5kdc[8988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:49 dist-test-slave-r238 krb5kdc[8988](info): set up 2 sockets
Sep 28 02:47:49 dist-test-slave-r238 krb5kdc[8988](info): commencing operation
krb5kdc: starting...
W20260928 02:47:50.156759 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:47:50 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/rangeradmin_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/rangeradmin_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/rangerlookup_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/rangerlookup_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
I20260928 02:47:50.331177 29666 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9u0ndH/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/postgres -l logfile start
2026-09-28 02:47:52.857 UTC [9018] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:47:52.857 UTC [9018] LOG: listening on IPv4 address "127.28.248.148", port 48531
2026-09-28 02:47:52.860 UTC [9018] LOG: listening on Unix socket "/tmp/.s.PGSQL.48531"
2026-09-28 02:47:52.881 UTC [9021] LOG: database system was shut down at 2026-09-28 02:47:51 UTC
2026-09-28 02:47:52.885 UTC [9018] LOG: database system is ready to accept connections
I20260928 02:47:53.836561 29666 mini_postgres.cc:96] Postgres bound to 48531
2026-09-28 02:47:53.843 UTC [9029] FATAL: database "slave" does not exist
127.28.248.148:48531 - accepting connections
I20260928 02:47:53.843648 29666 mini_ranger.cc:161] Starting Ranger...
I20260928 02:47:53.861017 29666 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:47:54.381618 29666 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:47:54.381724 29666 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-admin
2026-09-28 02:47:54,655 [I] DB FLAVOR :POSTGRES
2026-09-28 02:47:54,655 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:47:54,656 [I] Checking connection..
2026-09-28 02:47:54,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:47:54,885 [I] Checking connection passed.
2026-09-28 02:47:54,886 [I] --------- Verifying version history table ---------
2026-09-28 02:47:54,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:47:55,132 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:47:55,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:47:55,360 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:47:55,360 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:47:55,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:47:55.592 UTC [9093] WARNING: there is no transaction in progress
2026-09-28 02:47:55,598 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:47:55,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:47:55,832 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:47:55,832 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:47:55,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:47:56,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:47:56,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r238.c.gcp-upstream.internal','N') ;"
2026-09-28 02:47:56,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:47:56,829 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:47:56,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:47:57,090 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:47:57,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:47:57,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:47:57,567 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:47:57,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:47:58.562 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.579 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.597 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.614 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.631 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.653 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.716 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.723 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.738 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.749 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.760 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.775 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.786 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.792 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.799 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:58.806 UTC [9201] WARNING: there is no transaction in progress
2026-09-28 02:47:59,500 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:47:59,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:47:59,740 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:47:59,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:47:59,974 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:47:59,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:48:00,241 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:48:00,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:48:00,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:48:00,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r238.c.gcp-upstream.internal';"
2026-09-28 02:48:00,961 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:48:00,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:48:01,225 [I] DB_PATCHES have already been applied
I20260928 02:48:01.232501 29666 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-admin:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9u0ndH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:48:01.232565 29666 mini_ranger.cc:193] Using host: 127.28.248.148
I20260928 02:48:01.236537 29666 mini_ranger.cc:227] Ranger admin URL: http://127.28.248.148:52995
Sep 28, 2026 2:48:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:48:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:48:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:48:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:48:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:48:02.359648 29666 mini_ranger.cc:160] Time spent starting Ranger: real 8.516s user 0.004s sys 0.007s
Sep 28, 2026 2:48:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:48:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:48:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:48:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:48:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:48:22.902348 29666 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.248.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.248.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/rangerkms_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.248.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/rangerkms_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.248.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.248.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.248.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/HTTP_127.28.248.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:48:22 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563702, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:48:22.987012 29666 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:48:22.996232 29666 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:48:23.883385 29666 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:48:23.883486 29666 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms
I20260928 02:48:23.883504 29666 mini_ranger_kms.cc:227] Using postgres at 127.28.248.148:48531
2026-09-28 02:48:24,060 [I] DB FLAVOR :POSTGRES
2026-09-28 02:48:24,060 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:48:24,060 [I] Checking connection
2026-09-28 02:48:24,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:48:24,308 [I] connection success
2026-09-28 02:48:24,309 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:48:24,309 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:48:24,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:48:24,533 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:48:24,534 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:48:24,534 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:48:24,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.248.148:48531/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:48:24,804 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:48:25.231948 29666 mini_ranger_kms.cc:310] Created kms service
I20260928 02:48:25.440022 29666 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:48:25.488598 29666 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:48:25 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563702, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:25.870113 29666 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:48:25.870364 29666 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms:/tmp/dist-test-task9u0ndH/build/debug/bin/postgresql.jar:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9u0ndH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:48:25.870407 29666 mini_ranger_kms.cc:240] Using host: 127.28.248.148
I20260928 02:48:25.874367 29666 mini_ranger_kms.cc:276] Ranger KMS PID: 9381
I20260928 02:48:25.874470 29666 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.248.148:55467
Sep 28, 2026 2:48:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:48:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:48:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:48:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:48:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:48:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 02:48:27 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563707, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.248.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:28.141161 29666 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.154s user 0.015s sys 0.147s
I20260928 02:48:28.141299 29666 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:48:28.141366 29666 mini_ranger_kms.cc:401] 127.28.248.148:55467/kms/v1/keys
Sep 28 02:48:28 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563707, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.248.148@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
Sep 28 02:48:28 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563707, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.248.148@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
Sep 28 02:48:31 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:48:31.060206 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38031
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:38031
--ranger_config_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:31.177887 9432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:31.178195 9432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:31.178241 9432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:31.182293 9432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:31.182358 9432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:31.182376 9432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:31.182392 9432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:31.182408 9432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:31.187986 9432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.248.148:55467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:38031
--ranger_config_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:31.189123 9432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:31.190119 9432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:31.198873 9440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:31.198865 9437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:31.199153 9432 server_base.cc:1034] running on GCE node
W20260928 02:48:31.198865 9438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:31.199836 9432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:31.201133 9432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:31.202365 9432 hybrid_clock.cc:648] HybridClock initialized: now 1790563711202351 us; error 56 us; skew 500 ppm
Sep 28 02:48:31 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563711, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:31.209903 9432 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:31.210839 9432 webserver.cc:460] Webserver started at http://127.28.248.190:38005/ using document root <none> and password file <none>
I20260928 02:48:31.211381 9432 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:31.211433 9432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:31.211599 9432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:48:31 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563711, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:31.259013 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "5dc4004221d54fc2a5d9d7cc065cf15a"
format_stamp: "Formatted at 2026-09-28 02:48:31 on dist-test-slave-r238"
server_key: "2f06d926870638ac4bed425a70bded8a"
server_key_iv: "9eb58fe11b543dc35f3a59785ebfb251"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:31.259666 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "5dc4004221d54fc2a5d9d7cc065cf15a"
format_stamp: "Formatted at 2026-09-28 02:48:31 on dist-test-slave-r238"
server_key: "2f06d926870638ac4bed425a70bded8a"
server_key_iv: "9eb58fe11b543dc35f3a59785ebfb251"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:31.263863 9432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:48:31.282045 9447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:31.283785 9432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:48:31.283951 9432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "5dc4004221d54fc2a5d9d7cc065cf15a"
format_stamp: "Formatted at 2026-09-28 02:48:31 on dist-test-slave-r238"
server_key: "2f06d926870638ac4bed425a70bded8a"
server_key_iv: "9eb58fe11b543dc35f3a59785ebfb251"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:31.284106 9432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:31.311868 9432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:31.314535 9432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:31.314692 9432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:31.321880 9432 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:38031
I20260928 02:48:31.321919 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:38031 every 8 connection(s)
I20260928 02:48:31.322829 9432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:31.325604 9500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:31.328650 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 9432
I20260928 02:48:31.328783 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:48:31.329099 29666 external_mini_cluster.cc:1428] Setting key 052cf30cad2c128661c768705a97c7a0
I20260928 02:48:31.331521 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Bootstrap starting.
I20260928 02:48:31.334077 9500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:31.334990 9500 log.cc:824] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:31.337375 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: No bootstrap required, opened a new log
Sep 28 02:48:31 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:31.342172 9500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:31.342438 9500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:31.342525 9500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc4004221d54fc2a5d9d7cc065cf15a, State: Initialized, Role: FOLLOWER
I20260928 02:48:31.343004 9500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:31.343154 9500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:31.343246 9500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:31.343374 9500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:31.344336 9500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:31.344408 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:31.330514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.28.248.190:38031)
0928 02:48:31.330901 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:31.330916 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:31.330952 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:48:31.331626 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:31.331652 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:31.331987 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:31.332259 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:31.333210 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:31.334263 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:31.335415 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:31.336031 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:31.336153 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:31.339109 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:31.339116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:31.339134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:31.339165 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:31.341545 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:31.342080 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:31.342086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:31.342092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:31.342208 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:31.342498 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:31.342502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:31.342503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:31.342964 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:31.343189 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:31.343413 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:48:31.343698 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":120,"threads_started":1}
I20260928 02:48:31.344753 9500 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dc4004221d54fc2a5d9d7cc065cf15a; no voters:
I20260928 02:48:31.345095 9500 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:48:31.345198 9505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:31.345410 9505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 LEADER]: Becoming Leader. State: Replica: 5dc4004221d54fc2a5d9d7cc065cf15a, State: Running, Role: LEADER
I20260928 02:48:31.345798 9505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:31.346975 9505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc4004221d54fc2a5d9d7cc065cf15a. Latest consensus state: current_term: 1 leader_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } } }
I20260928 02:48:31.347111 9505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:31.347393 9506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } } }
I20260928 02:48:31.347509 9506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:31.347420 9500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:31.347573 9509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:31.350772 9509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:31.351212 9500 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:48:31.357522 9509 catalog_manager.cc:1347] Generated new cluster ID: 3e9bad8d088749918c408b1c1de8765c
I20260928 02:48:31.357609 9509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:31.377478 9509 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:31.379009 9509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:31.385773 9509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Generated new TSK 0
I20260928 02:48:31.386739 9509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:48:32.088023 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38031
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:32.247534 9545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:32.247885 9545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:32.247958 9545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:32.254024 9545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:32.254189 9545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:32.254334 9545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:48:32.262060 9545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.248.148:55467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:32.263748 9545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:32.265177 9545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:32.277750 9553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:32.277998 9555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:32.278293 9552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:32.279361 9545 server_base.cc:1034] running on GCE node
I20260928 02:48:32.279819 9545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:32.280580 9545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:32.281801 9545 hybrid_clock.cc:648] HybridClock initialized: now 1790563712281777 us; error 37 us; skew 500 ppm
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:32.285712 9545 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:48:32.286856 9545 webserver.cc:460] Webserver started at http://127.28.248.129:39267/ using document root <none> and password file <none>
I20260928 02:48:32.287580 9545 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:32.287711 9545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:32.287986 9545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:32.311072 9545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "1a42a9ae5e05427ea79dd63bc61623b0"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "3fe90462fb58dfdeaef35432c6eed94e"
server_key_iv: "9269ce7ddb8be48a64722dc404208fdf"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:32.312126 9545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "1a42a9ae5e05427ea79dd63bc61623b0"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "3fe90462fb58dfdeaef35432c6eed94e"
server_key_iv: "9269ce7ddb8be48a64722dc404208fdf"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:32.317761 9545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:48:32.350055 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:32.351866 9545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:48:32.352090 9545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "1a42a9ae5e05427ea79dd63bc61623b0"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "3fe90462fb58dfdeaef35432c6eed94e"
server_key_iv: "9269ce7ddb8be48a64722dc404208fdf"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:32.352324 9545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:32.392297 9545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:32.399631 9545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:32.399814 9545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:32.400513 9545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:32.401582 9545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:32.401635 9545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:32.401690 9545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:32.401726 9545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:32.412256 9545 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:39567
I20260928 02:48:32.413466 9545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:48:32.417608 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 9545
I20260928 02:48:32.417728 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:48:32.418031 29666 external_mini_cluster.cc:1428] Setting key 15c32e48d172f5f484d97e18ecc4f364
I20260928 02:48:32.418601 9677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:39567 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:32.447741 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:32.421927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:42915 (local address 127.28.248.190:38031)
0928 02:48:32.422550 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:32.422554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:32.424182 (+ 1628us) server_negotiation.cc:407] Connection header received
0928 02:48:32.424231 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:32.424234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:32.424288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:32.424372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:32.428970 (+ 4598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.429849 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.431692 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.432075 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.432165 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:32.443095 (+ 10930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:32.443101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:32.443104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:32.443136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:32.445121 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:32.445824 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:32.445828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:32.445830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:32.445922 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:32.446359 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:32.446362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:32.446364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:32.446618 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:32.446705 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:32.447396 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 02:48:32.447547 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":76,"threads_started":1}
I20260928 02:48:32.449697 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:32.420850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38031 (local address 127.28.248.129:42915)
0928 02:48:32.422822 (+ 1972us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:32.422876 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:32.423889 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:32.424713 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:32.424725 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:32.425191 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:32.425855 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:32.425877 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.430024 (+ 4147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.430028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:32.431526 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:32.431541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.432380 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.432384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:32.432628 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:32.437954 (+ 5326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:32.437979 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:32.442829 (+ 4850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:32.445284 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:32.445291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:32.445312 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:32.445701 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:32.446041 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:32.446044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:32.446047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:32.446246 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:32.446773 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:32.446780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:32.447116 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 02:48:32.448199 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0928 02:48:32.448578 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1760,"thread_start_us":132,"threads_started":1}
I20260928 02:48:32.451403 9678 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38031
I20260928 02:48:32.451668 9678 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:32.452246 9678 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:32.454069 9464 ts_manager.cc:194] Registered new tserver with Master: 1a42a9ae5e05427ea79dd63bc61623b0 (127.28.248.129:39567)
I20260928 02:48:32.455654 9464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:42915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:48:32.477936 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38031
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:48:32.638466 9684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:32.638823 9684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:32.638883 9684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:32.644513 9684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:32.644590 9684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:32.644690 9684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:48:32.652145 9684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.248.148:55467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:32.653527 9684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:32.654800 9684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:32.667308 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:32.667735 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:32.667884 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:32.669160 9684 server_base.cc:1034] running on GCE node
I20260928 02:48:32.669621 9684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:32.670385 9684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:32.671581 9684 hybrid_clock.cc:648] HybridClock initialized: now 1790563712671515 us; error 77 us; skew 500 ppm
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:32.674990 9684 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:48:32.675979 9684 webserver.cc:460] Webserver started at http://127.28.248.130:38539/ using document root <none> and password file <none>
I20260928 02:48:32.676645 9684 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:32.676721 9684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:32.676941 9684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:32.698064 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "7c63f45322f74a7886b44f89a2a963e9"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "2ce99c11cdd620c948858952ea575931"
server_key_iv: "ea27baf4da3f2fc844c283ed9a281438"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:32.698786 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "7c63f45322f74a7886b44f89a2a963e9"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "2ce99c11cdd620c948858952ea575931"
server_key_iv: "ea27baf4da3f2fc844c283ed9a281438"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:32.705006 9684 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:48:32.727100 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:32.728744 9684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:48:32.728866 9684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "7c63f45322f74a7886b44f89a2a963e9"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "2ce99c11cdd620c948858952ea575931"
server_key_iv: "ea27baf4da3f2fc844c283ed9a281438"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:32.728998 9684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:32.749517 9684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:32.753000 9684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:32.753180 9684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:32.753785 9684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:32.754735 9684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:32.754791 9684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:32.754845 9684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:32.754875 9684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:32.767647 9684 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:41557
I20260928 02:48:32.768783 9684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:48:32.769287 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:41557 every 8 connection(s)
I20260928 02:48:32.771620 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 9684
I20260928 02:48:32.771726 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:48:32.772055 29666 external_mini_cluster.cc:1428] Setting key 06c3b63be7fc0ae362afa378c07d731b
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:32.791288 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:32.773221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:32823 (local address 127.28.248.190:38031)
0928 02:48:32.773369 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:32.773374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:32.774356 (+ 982us) server_negotiation.cc:407] Connection header received
0928 02:48:32.775503 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:32.775506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:32.775554 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:32.775604 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:32.777396 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.778137 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.779823 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.780171 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.780241 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:32.784396 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:32.784400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:32.784403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:32.784431 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:32.786546 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:32.788599 (+ 2053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:32.788630 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:32.788632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:32.788715 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:32.789900 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:32.789904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:32.789906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:32.790149 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:32.790244 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:32.790945 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 02:48:32.791109 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:48:32.792922 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:32.772000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38031 (local address 127.28.248.130:32823)
0928 02:48:32.774201 (+ 2201us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:32.774256 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:32.775243 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:32.775986 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:32.775994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:32.776345 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:32.777230 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:32.777247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.778307 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.778310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:32.779687 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:32.779697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:32.780460 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:32.780464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:32.780692 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:32.782522 (+ 1830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:32.782543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:32.784176 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:32.788041 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:32.788048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:32.788067 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:32.788489 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:32.788819 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:32.788822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:32.788824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:32.789039 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:32.790352 (+ 1313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:32.790359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:32.790679 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 02:48:32.791518 (+ 839us) client_negotiation.cc:240] Negotiation successful
0928 02:48:32.791868 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1995,"thread_start_us":131,"threads_started":1}
I20260928 02:48:32.794163 9814 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38031
I20260928 02:48:32.794417 9814 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:32.794961 9814 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:32.796021 9464 ts_manager.cc:194] Registered new tserver with Master: 7c63f45322f74a7886b44f89a2a963e9 (127.28.248.130:41557)
I20260928 02:48:32.796773 9464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:32823
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:48:32.820640 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38031
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:32.965957 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:32.966316 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:32.966384 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:32.972539 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:32.972640 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:32.972750 9819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:48:32.980952 9819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.248.148:55467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:32.982385 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:32.983747 9819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:32.996882 9825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 28 02:48:32 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563712, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
W20260928 02:48:32.997761 9824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:32.998267 9819 server_base.cc:1034] running on GCE node
W20260928 02:48:32.998574 9827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:32.999076 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:32.999815 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:33.005972 9819 hybrid_clock.cc:648] HybridClock initialized: now 1790563713005881 us; error 99 us; skew 500 ppm
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:33.009828 9819 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:48:33.010957 9819 webserver.cc:460] Webserver started at http://127.28.248.131:34211/ using document root <none> and password file <none>
I20260928 02:48:33.011715 9819 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:33.011813 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:33.012065 9819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:33.047011 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "d766f00faac14f6f936cba75adf36351"
format_stamp: "Formatted at 2026-09-28 02:48:33 on dist-test-slave-r238"
server_key: "0f8a30c9c509bfaca48c44c2c3c7bc89"
server_key_iv: "4f91bb954a2f6a785db70aa61490d506"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:33.047840 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "d766f00faac14f6f936cba75adf36351"
format_stamp: "Formatted at 2026-09-28 02:48:33 on dist-test-slave-r238"
server_key: "0f8a30c9c509bfaca48c44c2c3c7bc89"
server_key_iv: "4f91bb954a2f6a785db70aa61490d506"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:33.063417 9819 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260928 02:48:33.093273 9834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.094884 9819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:48:33.095011 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "d766f00faac14f6f936cba75adf36351"
format_stamp: "Formatted at 2026-09-28 02:48:33 on dist-test-slave-r238"
server_key: "0f8a30c9c509bfaca48c44c2c3c7bc89"
server_key_iv: "4f91bb954a2f6a785db70aa61490d506"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:33.095135 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:33.127456 9819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:33.131114 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:33.131359 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:33.132038 9819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:33.133059 9819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:33.133164 9819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.133256 9819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:33.133315 9819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.151288 9819 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:35229
I20260928 02:48:33.152572 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:48:33.153194 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 9819
I20260928 02:48:33.153324 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:48:33.153621 29666 external_mini_cluster.cc:1428] Setting key 25a01ae3ef2395868ea66ee8e9ed96a3
I20260928 02:48:33.156647 9947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:35229 every 8 connection(s)
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:33.178210 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:33.160340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:38269 (local address 127.28.248.190:38031)
0928 02:48:33.160877 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:33.160882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:33.161727 (+ 845us) server_negotiation.cc:407] Connection header received
0928 02:48:33.163094 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:33.163098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:33.163159 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:33.163217 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:33.165446 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.166313 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.169631 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.169948 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.170021 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:33.173792 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:33.173796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:33.173798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:33.173827 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:33.175669 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:33.176286 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:33.176290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:33.176293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:33.176374 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:33.176852 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:33.176856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:33.176857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:33.177079 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:33.177167 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:33.177887 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 02:48:33.178042 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:48:33.182011 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:33.159126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38031 (local address 127.28.248.131:38269)
0928 02:48:33.161569 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:33.161628 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:33.162831 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:33.164062 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:33.164073 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:33.164496 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:33.165263 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:33.165288 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.168041 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.168047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:33.169494 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:33.169505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.170221 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.170225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:33.170424 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:33.171159 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:33.171179 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:33.173605 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:33.175807 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:33.175813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:33.175831 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:33.176171 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:33.176485 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:33.176488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:33.176490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:33.176699 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:33.177236 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:33.177242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:33.177605 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 02:48:33.180202 (+ 2597us) client_negotiation.cc:240] Negotiation successful
0928 02:48:33.180579 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2191,"thread_start_us":154,"threads_started":1}
I20260928 02:48:33.183633 9948 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38031
I20260928 02:48:33.183869 9948 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:33.184418 9948 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:33.185621 9464 ts_manager.cc:194] Registered new tserver with Master: d766f00faac14f6f936cba75adf36351 (127.28.248.131:35229)
I20260928 02:48:33.186348 9464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:38269
I20260928 02:48:33.191725 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:48:33.192710 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 9545
I20260928 02:48:33.200080 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 9684
I20260928 02:48:33.208967 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 9819
I20260928 02:48:33.217298 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 9432
I20260928 02:48:33.227779 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38031
--webserver_interface=127.28.248.190
--webserver_port=38005
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:38031
--ranger_config_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:33.339447 9951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:33.339712 9951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:33.339751 9951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:33.343999 9951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:33.344066 9951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:33.344085 9951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:33.344102 9951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:33.344118 9951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:33.350696 9951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:38031
--ranger_config_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=38005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:33.352334 9951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:33.353538 9951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:33.362151 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:33.362174 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:33.362397 9951 server_base.cc:1034] running on GCE node
W20260928 02:48:33.362499 9959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:33.362987 9951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:33.364189 9951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:33.365378 9951 hybrid_clock.cc:648] HybridClock initialized: now 1790563713365352 us; error 53 us; skew 500 ppm
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:33.368171 9951 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:33.369192 9951 webserver.cc:460] Webserver started at http://127.28.248.190:38005/ using document root <none> and password file <none>
I20260928 02:48:33.369724 9951 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:33.369789 9951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:33.425351 9951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:48:33.428161 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.429801 9951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:48:33.429942 9951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "5dc4004221d54fc2a5d9d7cc065cf15a"
format_stamp: "Formatted at 2026-09-28 02:48:31 on dist-test-slave-r238"
server_key: "2f06d926870638ac4bed425a70bded8a"
server_key_iv: "9eb58fe11b543dc35f3a59785ebfb251"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:33.430688 9951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:33.463539 9951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:33.467120 9951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:33.467325 9951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:33.476109 9951 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:38031
I20260928 02:48:33.476119 10019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:38031 every 8 connection(s)
I20260928 02:48:33.477612 9951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:33.481877 10020 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:48:33.483444 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Bootstrap starting.
I20260928 02:48:33.485473 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 9951
I20260928 02:48:33.485980 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:39567
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:38031
--webserver_port=39267
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:33.500413 10020 log.cc:824] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:33.507529 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:48:33.507992 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Bootstrap complete.
I20260928 02:48:33.512387 10020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:33.512892 10020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc4004221d54fc2a5d9d7cc065cf15a, State: Initialized, Role: FOLLOWER
I20260928 02:48:33.513449 10020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:33.513607 10020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:33.513679 10020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:33.513799 10020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:48:33.515729 10020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:33.516086 10020 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dc4004221d54fc2a5d9d7cc065cf15a; no voters:
I20260928 02:48:33.516486 10020 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:48:33.516654 10024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:48:33.516925 10024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [term 2 LEADER]: Becoming Leader. State: Replica: 5dc4004221d54fc2a5d9d7cc065cf15a, State: Running, Role: LEADER
I20260928 02:48:33.517285 10024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } }
I20260928 02:48:33.517815 10020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:33.518394 10024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } } }
I20260928 02:48:33.518589 10024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:33.518886 10024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc4004221d54fc2a5d9d7cc065cf15a. Latest consensus state: current_term: 2 leader_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc4004221d54fc2a5d9d7cc065cf15a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38031 } } }
I20260928 02:48:33.519048 10024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:33.519388 10030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:33.523494 10030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:33.524088 10020 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:48:33.524376 10030 catalog_manager.cc:1259] Loaded cluster ID: 3e9bad8d088749918c408b1c1de8765c
I20260928 02:48:33.524480 10030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:33.526897 10030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:33.527716 10030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Loaded TSK: 0
I20260928 02:48:33.529030 10030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:48:33.640504 10022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:33.640884 10022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:33.640947 10022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:33.647281 10022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:33.647359 10022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:33.647450 10022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:48:33.654870 10022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:39567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=39267
--tserver_master_addrs=127.28.248.190:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:33.656282 10022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:33.657678 10022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:33.670766 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:33.670766 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:33.671630 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:33.672137 10022 server_base.cc:1034] running on GCE node
I20260928 02:48:33.672633 10022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:33.673412 10022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:33.674602 10022 hybrid_clock.cc:648] HybridClock initialized: now 1790563713674574 us; error 43 us; skew 500 ppm
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:33.678084 10022 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:48:33.679081 10022 webserver.cc:460] Webserver started at http://127.28.248.129:39267/ using document root <none> and password file <none>
I20260928 02:48:33.679759 10022 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:33.679843 10022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:33.700583 10022 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:48:33.703272 10068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.704916 10022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:48:33.705030 10022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "1a42a9ae5e05427ea79dd63bc61623b0"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "3fe90462fb58dfdeaef35432c6eed94e"
server_key_iv: "9269ce7ddb8be48a64722dc404208fdf"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:33.705720 10022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:33.737876 10022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:33.741748 10022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:33.742066 10022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:33.742808 10022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:33.743880 10022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:33.743989 10022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.744114 10022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:33.744190 10022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:33.757256 10022 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:39567
I20260928 02:48:33.757596 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:39567 every 8 connection(s)
I20260928 02:48:33.758654 10022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:48:33.760134 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10022
I20260928 02:48:33.760593 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:41557
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:38031
--webserver_port=38539
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:33.781620 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:33.761447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38031 (local address 127.28.248.129:55153)
0928 02:48:33.764047 (+ 2600us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:33.764105 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:33.765216 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:33.767114 (+ 1898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:33.767122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:33.767480 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:33.768095 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:33.768116 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.770057 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.770060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:33.771462 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:33.771473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.772750 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.772754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:33.772949 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:33.773647 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:33.773670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:33.775231 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:33.777806 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:33.777813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:33.777828 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:33.778179 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:33.778621 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:33.778624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:33.778627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:33.778801 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:33.779628 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:33.779634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:33.779939 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 02:48:33.780140 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 02:48:33.780496 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2375,"thread_start_us":155,"threads_started":1}
I20260928 02:48:33.783461 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:33.762916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:55153 (local address 127.28.248.190:38031)
0928 02:48:33.763527 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:33.763548 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:33.765549 (+ 2001us) server_negotiation.cc:407] Connection header received
0928 02:48:33.765837 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:33.765851 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:33.766271 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:33.766622 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:33.768283 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.769889 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.771601 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:33.772325 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:33.772471 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:33.775468 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:33.775478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:33.775501 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:33.775533 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:33.777669 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:33.778322 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:33.778340 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:33.778347 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:33.778499 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:33.778918 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:33.778921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:33.778923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:33.779432 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:33.779544 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:33.782196 (+ 2652us) server_negotiation.cc:299] Negotiation successful
0928 02:48:33.782527 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":108,"threads_started":1}
I20260928 02:48:33.786854 10182 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38031
I20260928 02:48:33.787092 10182 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:33.787688 10182 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
W20260928 02:48:33.790109 10182 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:38031 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:48:33.790232 10182 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:33.790431 10182 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:33.791405 10182 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:33.791569 10182 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
W20260928 02:48:33.792404 10182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:48:33.919225 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:33.919586 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:33.919655 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:33.925640 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:33.925720 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:33.925803 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:48:33.933430 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:41557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=38539
--tserver_master_addrs=127.28.248.190:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:33.934846 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:33.936182 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:33.948843 10192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:33.949271 10194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:33.955698 10185 server_base.cc:1034] running on GCE node
W20260928 02:48:33.956980 10191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:33.961962 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:33.962891 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:33.966612 10185 hybrid_clock.cc:648] HybridClock initialized: now 1790563713966393 us; error 242 us; skew 500 ppm
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:33.970559 10185 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:48:33.971946 10185 webserver.cc:460] Webserver started at http://127.28.248.130:38539/ using document root <none> and password file <none>
I20260928 02:48:33.972790 10185 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:33.972954 10185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:48:33 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:34.008770 10185 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.000s
I20260928 02:48:34.013468 10202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:34.015190 10185 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 02:48:34.015328 10185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "7c63f45322f74a7886b44f89a2a963e9"
format_stamp: "Formatted at 2026-09-28 02:48:32 on dist-test-slave-r238"
server_key: "2ce99c11cdd620c948858952ea575931"
server_key_iv: "ea27baf4da3f2fc844c283ed9a281438"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:34.016085 10185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:34.039125 10185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:34.042850 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:34.043103 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:34.043778 10185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:34.044859 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:34.044963 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:34.045058 10185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:34.045120 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:34.056023 10185 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:41557
I20260928 02:48:34.057545 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:48:34.065749 10315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:41557 every 8 connection(s)
I20260928 02:48:34.066394 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10185
I20260928 02:48:34.066861 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.248.148:55467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:35229
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:38031
--webserver_port=34211
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:48:34 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:34.076323 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:34.061072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:36209 (local address 127.28.248.190:38031)
0928 02:48:34.062228 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:34.062234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:34.062358 (+ 124us) server_negotiation.cc:407] Connection header received
0928 02:48:34.063601 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:34.063605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:34.063664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:34.063720 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:34.065431 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.066267 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.068388 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.068780 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.068878 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:34.071991 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:34.071995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:34.071998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:34.072026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:34.073868 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:34.074507 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:34.074510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:34.074512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:34.074598 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:34.075015 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:34.075018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:34.075021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:34.075237 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:34.075305 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:34.075996 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 02:48:34.076132 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070}
I20260928 02:48:34.078229 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:34.060479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38031 (local address 127.28.248.130:36209)
0928 02:48:34.062202 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:34.062258 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:34.063347 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:34.064121 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:34.064130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:34.064505 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:34.065248 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:34.065270 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.066557 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.066560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:34.068241 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:34.068252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.069089 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.069093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:34.069297 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:34.069980 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:34.070004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:34.071766 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:34.074021 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:34.074028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:34.074046 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:34.074387 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:34.074708 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:34.074711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:34.074713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:34.074902 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:34.075386 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:34.075393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:34.075706 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 02:48:34.076813 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0928 02:48:34.077173 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"thread_start_us":133,"threads_started":1}
I20260928 02:48:34.080111 10316 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38031
I20260928 02:48:34.080361 10316 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.081100 10316 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
W20260928 02:48:34.082825 10316 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:38031 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:48:34.082962 10316 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.083129 10316 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:34.083971 10316 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.084120 10316 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
W20260928 02:48:34.084904 10316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:48:34.242825 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:34.243312 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:34.243458 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:34.248335 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:34.248426 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:34.248541 10319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:48:34.254194 10319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.248.148:55467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:35229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=34211
--tserver_master_addrs=127.28.248.190:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:34.255556 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:34.256667 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:34.274727 10328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:34.274727 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:34.275360 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:34.276357 10319 server_base.cc:1034] running on GCE node
I20260928 02:48:34.276909 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:34.277633 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:34.278821 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790563714278774 us; error 50 us; skew 500 ppm
Sep 28 02:48:34 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:34.282331 10319 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:48:34.283685 10319 webserver.cc:460] Webserver started at http://127.28.248.131:34211/ using document root <none> and password file <none>
I20260928 02:48:34.284269 10319 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:34.284382 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:48:34 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:34.302268 10319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:48:34.307639 10336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:34.309337 10319 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260928 02:48:34.309461 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "d766f00faac14f6f936cba75adf36351"
format_stamp: "Formatted at 2026-09-28 02:48:33 on dist-test-slave-r238"
server_key: "0f8a30c9c509bfaca48c44c2c3c7bc89"
server_key_iv: "4f91bb954a2f6a785db70aa61490d506"
server_key_version: "kuduclusterkey@0"
I20260928 02:48:34.310160 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:34.328935 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:34.332720 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:34.332924 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:34.333664 10319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:34.334681 10319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:34.334744 10319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:34.334801 10319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:34.334829 10319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:34.348436 10319 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:35229
I20260928 02:48:34.349741 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:48:34.350602 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:35229 every 8 connection(s)
I20260928 02:48:34.357134 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10319
Sep 28 02:48:34 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:34.369942 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:34.354740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:48979 (local address 127.28.248.190:38031)
0928 02:48:34.354866 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:34.354871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:34.355392 (+ 521us) server_negotiation.cc:407] Connection header received
0928 02:48:34.356881 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:34.356885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:34.356942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:34.356995 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:34.358978 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.359954 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.361676 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.362026 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.362100 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:34.365680 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:34.365684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:34.365687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:34.365712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:34.367500 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:34.368132 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:34.368135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:34.368137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:34.368221 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:34.368657 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:34.368661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:34.368663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:34.368876 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:34.368956 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:34.369644 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 02:48:34.369774 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:48:34.371616 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:34.353575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38031 (local address 127.28.248.131:48979)
0928 02:48:34.355213 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:34.355271 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:34.356361 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:34.357417 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:34.357425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:34.358127 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:34.358802 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:34.358827 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.360112 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.360116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:34.361535 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:34.361546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.362400 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.362404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:34.362606 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:34.363531 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:34.363553 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:34.365484 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:34.367643 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:34.367650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:34.367668 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:34.368013 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:34.368335 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:34.368338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:34.368340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:34.368514 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:34.369023 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:34.369029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:34.369366 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 02:48:34.370177 (+ 811us) client_negotiation.cc:240] Negotiation successful
0928 02:48:34.370547 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"thread_start_us":158,"threads_started":1}
I20260928 02:48:34.372965 10450 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38031
I20260928 02:48:34.373215 10450 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.373795 10450 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
W20260928 02:48:34.375444 10450 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:38031 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:48:34.375571 10450 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.375746 10450 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:34.376510 10450 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.376703 10450 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
W20260928 02:48:34.377394 10450 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:48:34.393116 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:34.357812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.28.248.190:38031)
0928 02:48:34.364401 (+ 6589us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:34.364412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:34.364922 (+ 510us) server_negotiation.cc:407] Connection header received
0928 02:48:34.364967 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:34.364970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:34.365017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:34.365093 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:34.378495 (+ 13402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.379273 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.380546 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:34.380951 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:34.381089 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:34.382928 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:34.382932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:34.382935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:34.382968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:34.384923 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:34.387046 (+ 2123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:34.387050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:34.387052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:34.387149 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:34.387512 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:34.387516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:34.387517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:34.387767 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:34.388001 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:34.392775 (+ 4774us) server_negotiation.cc:299] Negotiation successful
0928 02:48:34.392922 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6484,"thread_start_us":107,"threads_started":1}
I20260928 02:48:34.792704 10182 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:34.793045 10182 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:34.794363 9984 ts_manager.cc:194] Registered new tserver with Master: 1a42a9ae5e05427ea79dd63bc61623b0 (127.28.248.129:39567)
I20260928 02:48:34.796114 9984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:55153
Sep 28 02:48:34 dist-test-slave-r238 krb5kdc[8988](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:35.085136 10316 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:35.085448 10316 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:35.086213 9984 ts_manager.cc:194] Registered new tserver with Master: 7c63f45322f74a7886b44f89a2a963e9 (127.28.248.130:41557)
I20260928 02:48:35.086962 9984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:36209
Sep 28 02:48:35 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:35.377611 10450 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:35.377892 10450 heartbeater.cc:511] Master 127.28.248.190:38031 requested a full tablet report, sending...
I20260928 02:48:35.378685 9984 ts_manager.cc:194] Registered new tserver with Master: d766f00faac14f6f936cba75adf36351 (127.28.248.131:35229)
I20260928 02:48:35.379504 9984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:48979
I20260928 02:48:35.380592 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:48:35.391582 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.382165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.28.248.190:38031)
0928 02:48:35.382398 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.382402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.382614 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:48:35.382709 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.382712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.382762 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.382868 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:35.383651 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.384502 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.385672 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.386026 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.386135 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.387765 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:35.387769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:35.387772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:35.387797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:35.389737 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:35.390242 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:35.390246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:35.390248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:35.390347 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:35.390641 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:35.390645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:35.390646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:35.390867 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:35.391024 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.391257 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.391389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260928 02:48:35.395609 9984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:48:35.400331 9984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:48:35.421180 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.411711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39567 (local address 127.0.0.1:59190)
0928 02:48:35.414038 (+ 2327us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:35.414088 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:35.414282 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:35.414963 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:35.414969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:35.414990 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:35.415105 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.415111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.417284 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.417287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.418632 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.418641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.420578 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.420581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.420767 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.420803 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:48:35.420908 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:48:35.421022 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260928 02:48:35.421571 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.411819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.28.248.129:39567)
0928 02:48:35.412813 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.412820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.414191 (+ 1371us) server_negotiation.cc:407] Connection header received
0928 02:48:35.414447 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.414452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.414652 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.414794 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:35.415227 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.417169 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.419429 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.420246 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.420367 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.420514 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.421348 (+ 834us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.421451 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":112,"threads_started":1}
I20260928 02:48:35.421852 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.413521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.28.248.131:35229)
0928 02:48:35.413811 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.413817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.415195 (+ 1378us) server_negotiation.cc:407] Connection header received
0928 02:48:35.415989 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.415994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.416162 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.416304 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:35.416920 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.418005 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.419494 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.420383 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.420517 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.420665 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.421109 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.421204 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260928 02:48:35.422154 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.413138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:35229 (local address 127.0.0.1:42776)
0928 02:48:35.414942 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:35.414956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:35.415050 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:35.416685 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:35.416688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:35.416707 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:35.416802 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.416807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.418140 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.418143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.419341 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.419352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.420735 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.420739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.420833 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.420861 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:48:35.420937 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:48:35.421028 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1748,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260928 02:48:35.424790 10116 tablet_service.cc:1467] Processing CreateTablet for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 (DEFAULT_TABLE table=test-table [id=4dbe5de8a08243df8e0c93aceef2eb4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:35.425992 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:35.429220 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.413246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.28.248.130:41557)
0928 02:48:35.414597 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.414604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.415151 (+ 547us) server_negotiation.cc:407] Connection header received
0928 02:48:35.415497 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.415502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.415692 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.415833 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:35.417164 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.418292 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.419617 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.420338 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.420449 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.420542 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.428979 (+ 8437us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.429091 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1255,"thread_start_us":53,"threads_started":1}
I20260928 02:48:35.429155 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.412865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:41557 (local address 127.0.0.1:40182)
0928 02:48:35.414724 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:35.414740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:35.415329 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:35.416917 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:35.416920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:35.416931 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:35.417006 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.417011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.418430 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.418434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.419498 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.419506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.420586 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.420589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.428786 (+ 8197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.428823 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:48:35.428934 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:48:35.429020 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"mutex_wait_us":8052,"thread_start_us":84,"threads_started":1}
I20260928 02:48:35.436983 10483 tablet_bootstrap.cc:492] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: Bootstrap starting.
I20260928 02:48:35.438717 10250 tablet_service.cc:1467] Processing CreateTablet for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 (DEFAULT_TABLE table=test-table [id=4dbe5de8a08243df8e0c93aceef2eb4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:35.439657 10483 tablet_bootstrap.cc:654] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:35.439889 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:35.440744 10483 log.cc:824] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:35.441565 10384 tablet_service.cc:1467] Processing CreateTablet for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 (DEFAULT_TABLE table=test-table [id=4dbe5de8a08243df8e0c93aceef2eb4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:35.442586 10384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:35.451071 10483 tablet_bootstrap.cc:492] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: No bootstrap required, opened a new log
I20260928 02:48:35.451357 10483 ts_tablet_manager.cc:1397] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260928 02:48:35.455469 10483 raft_consensus.cc:348] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.455899 10483 raft_consensus.cc:374] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:35.456003 10483 raft_consensus.cc:729] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a42a9ae5e05427ea79dd63bc61623b0, State: Initialized, Role: FOLLOWER
I20260928 02:48:35.456598 10483 consensus_queue.cc:260] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.457823 10483 ts_tablet_manager.cc:1428] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:48:35.458209 10182 heartbeater.cc:503] Master 127.28.248.190:38031 was elected leader, sending a full tablet report...
I20260928 02:48:35.461449 10487 tablet_bootstrap.cc:492] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Bootstrap starting.
I20260928 02:48:35.462059 10485 tablet_bootstrap.cc:492] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Bootstrap starting.
I20260928 02:48:35.463873 10487 tablet_bootstrap.cc:654] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:35.464237 10485 tablet_bootstrap.cc:654] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:35.464947 10487 log.cc:824] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:35.465734 10485 log.cc:824] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:35.468422 10487 tablet_bootstrap.cc:492] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: No bootstrap required, opened a new log
I20260928 02:48:35.468645 10487 ts_tablet_manager.cc:1397] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:48:35.469002 10485 tablet_bootstrap.cc:492] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: No bootstrap required, opened a new log
I20260928 02:48:35.469167 10485 ts_tablet_manager.cc:1397] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:48:35.473047 10487 raft_consensus.cc:348] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.473323 10487 raft_consensus.cc:374] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:35.473385 10487 raft_consensus.cc:729] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d766f00faac14f6f936cba75adf36351, State: Initialized, Role: FOLLOWER
I20260928 02:48:35.473883 10487 consensus_queue.cc:260] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.473878 10485 raft_consensus.cc:348] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.474243 10485 raft_consensus.cc:374] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:35.474337 10485 raft_consensus.cc:729] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c63f45322f74a7886b44f89a2a963e9, State: Initialized, Role: FOLLOWER
I20260928 02:48:35.474852 10485 consensus_queue.cc:260] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.474926 10487 ts_tablet_manager.cc:1428] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:48:35.475281 10450 heartbeater.cc:503] Master 127.28.248.190:38031 was elected leader, sending a full tablet report...
I20260928 02:48:35.476989 10485 ts_tablet_manager.cc:1428] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:48:35.477342 10316 heartbeater.cc:503] Master 127.28.248.190:38031 was elected leader, sending a full tablet report...
W20260928 02:48:35.520406 10183 tablet.cc:2369] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:48:35 dist-test-slave-r238 krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for HTTP/127.28.248.148@KRBTEST.COM
I20260928 02:48:35.548339 10490 raft_consensus.cc:482] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:35.548578 10490 raft_consensus.cc:504] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.551105 10490 leader_election.cc:288] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a42a9ae5e05427ea79dd63bc61623b0 (127.28.248.129:39567), 7c63f45322f74a7886b44f89a2a963e9 (127.28.248.130:41557)
I20260928 02:48:35.556710 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.550234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39567 (local address 127.28.248.131:34487)
0928 02:48:35.551385 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:35.551398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:35.551874 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:35.552528 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:35.552531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:35.552551 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:35.552698 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.552704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.554011 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.554014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.555263 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.555274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.556236 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.556239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.556341 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.556368 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:48:35.556447 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:48:35.556517 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":296,"thread_start_us":85,"threads_started":1}
I20260928 02:48:35.556708 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.550402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:34487 (local address 127.28.248.129:39567)
0928 02:48:35.550533 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.550540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.551800 (+ 1260us) server_negotiation.cc:407] Connection header received
0928 02:48:35.552006 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.552010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.552058 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.552102 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:35.552831 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.553582 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.555495 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.556108 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.556229 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.556280 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.556480 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.556550 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:48:35.557591 10120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd21a706ce9f4dfe9f8a3d5eaa0952b5" candidate_uuid: "d766f00faac14f6f936cba75adf36351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" is_pre_election: true
I20260928 02:48:35.557930 10120 raft_consensus.cc:2454] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d766f00faac14f6f936cba75adf36351 in term 0.
I20260928 02:48:35.558544 10338 leader_election.cc:302] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a42a9ae5e05427ea79dd63bc61623b0, d766f00faac14f6f936cba75adf36351; no voters:
I20260928 02:48:35.558852 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.550642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:46833 (local address 127.28.248.130:41557)
0928 02:48:35.550747 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.550751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.550916 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:48:35.552147 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.552151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.552201 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.552249 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:35.553982 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.554617 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.557118 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.557638 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.557744 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.557798 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.558646 (+ 848us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.558734 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:48:35.558894 10490 raft_consensus.cc:2781] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:48:35.558974 10490 raft_consensus.cc:482] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:48:35.559011 10490 raft_consensus.cc:3037] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:35.559723 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.550230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:41557 (local address 127.28.248.131:46833)
0928 02:48:35.550835 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:35.550849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:35.551025 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:35.552388 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:35.552391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:35.552410 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:35.552989 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.552997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.555170 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.555174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.556683 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:35.556694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.558106 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.558109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:35.558372 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.558396 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:48:35.559041 (+ 645us) client_negotiation.cc:240] Negotiation successful
0928 02:48:35.559581 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":105,"threads_started":1}
I20260928 02:48:35.560267 10490 raft_consensus.cc:504] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.560591 10490 leader_election.cc:288] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [CANDIDATE]: Term 1 election: Requested vote from peers 1a42a9ae5e05427ea79dd63bc61623b0 (127.28.248.129:39567), 7c63f45322f74a7886b44f89a2a963e9 (127.28.248.130:41557)
I20260928 02:48:35.561424 10270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd21a706ce9f4dfe9f8a3d5eaa0952b5" candidate_uuid: "d766f00faac14f6f936cba75adf36351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c63f45322f74a7886b44f89a2a963e9" is_pre_election: true
I20260928 02:48:35.561707 10270 raft_consensus.cc:2454] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d766f00faac14f6f936cba75adf36351 in term 0.
I20260928 02:48:35.561990 10269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd21a706ce9f4dfe9f8a3d5eaa0952b5" candidate_uuid: "d766f00faac14f6f936cba75adf36351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c63f45322f74a7886b44f89a2a963e9"
I20260928 02:48:35.562076 10269 raft_consensus.cc:3037] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:35.563254 10269 raft_consensus.cc:2454] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d766f00faac14f6f936cba75adf36351 in term 1.
I20260928 02:48:35.563939 10337 leader_election.cc:302] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c63f45322f74a7886b44f89a2a963e9, d766f00faac14f6f936cba75adf36351; no voters:
I20260928 02:48:35.564530 10120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd21a706ce9f4dfe9f8a3d5eaa0952b5" candidate_uuid: "d766f00faac14f6f936cba75adf36351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a42a9ae5e05427ea79dd63bc61623b0"
I20260928 02:48:35.564674 10120 raft_consensus.cc:3037] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:35.564750 10490 raft_consensus.cc:2781] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:35.564962 10490 raft_consensus.cc:686] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 1 LEADER]: Becoming Leader. State: Replica: d766f00faac14f6f936cba75adf36351, State: Running, Role: LEADER
I20260928 02:48:35.565297 10490 consensus_queue.cc:237] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } }
I20260928 02:48:35.565945 10120 raft_consensus.cc:2454] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d766f00faac14f6f936cba75adf36351 in term 1.
W20260928 02:48:35.567271 10317 tablet.cc:2369] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:48:35.568567 9983 catalog_manager.cc:5495] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 reported cstate change: term changed from 0 to 1, leader changed from <none> to d766f00faac14f6f936cba75adf36351 (127.28.248.131). New cstate: current_term: 1 leader_uuid: "d766f00faac14f6f936cba75adf36351" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d766f00faac14f6f936cba75adf36351" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 35229 } health_report { overall_health: HEALTHY } } }
W20260928 02:48:35.602159 10451 tablet.cc:2369] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:48:35.612316 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.606833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.28.248.131:35229)
0928 02:48:35.606992 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:35.606996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:35.607011 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:48:35.607092 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:35.607095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:35.607157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:35.607272 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:35.607579 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.608405 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.609698 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:35.610057 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:35.610171 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:35.610539 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:35.611161 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:35.611278 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:35.612066 (+ 788us) server_negotiation.cc:299] Negotiation successful
0928 02:48:35.612152 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:48:35.620455 10269 raft_consensus.cc:1261] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 1 FOLLOWER]: Refusing update from remote peer d766f00faac14f6f936cba75adf36351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:35.621344 10490 consensus_queue.cc:1035] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c63f45322f74a7886b44f89a2a963e9" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:35.628959 10120 raft_consensus.cc:1261] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 1a42a9ae5e05427ea79dd63bc61623b0 [term 1 FOLLOWER]: Refusing update from remote peer d766f00faac14f6f936cba75adf36351: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:35.629703 10490 consensus_queue.cc:1035] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a42a9ae5e05427ea79dd63bc61623b0" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:35.643985 10500 mvcc.cc:205] Tried to move back new op lower bound from 7334148979173371904 to 7334148978958442496. Current Snapshot: MvccSnapshot[applied={T|T < 7334148979173371904}]
I20260928 02:48:35.649842 10502 mvcc.cc:205] Tried to move back new op lower bound from 7334148979173371904 to 7334148978958442496. Current Snapshot: MvccSnapshot[applied={T|T < 7334148979173371904 or (T in {7334148979173371904})}]
I20260928 02:48:35.650184 10499 mvcc.cc:205] Tried to move back new op lower bound from 7334148979173371904 to 7334148978958442496. Current Snapshot: MvccSnapshot[applied={T|T < 7334148979173371904 or (T in {7334148979173371904})}]
I20260928 02:48:35.652804 9983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:48:35.652982 9983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:48:35.655905 9983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dc4004221d54fc2a5d9d7cc065cf15a: Sending DeleteTablet for 3 replicas of tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5
I20260928 02:48:35.656723 10116 tablet_service.cc:1514] Processing DeleteTablet for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:35 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:59190
I20260928 02:48:35.657131 10384 tablet_service.cc:1514] Processing DeleteTablet for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:35 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:42776
I20260928 02:48:35.657478 10250 tablet_service.cc:1514] Processing DeleteTablet for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:35 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:40182
I20260928 02:48:35.658506 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10022
I20260928 02:48:35.670866 10510 tablet_replica.cc:330] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: stopping tablet replica
I20260928 02:48:35.671232 10509 tablet_replica.cc:330] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: stopping tablet replica
I20260928 02:48:35.671679 10509 raft_consensus.cc:2229] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:48:35.672088 10510 raft_consensus.cc:2229] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:48:35.673018 10509 raft_consensus.cc:2258] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:35.673419 10510 raft_consensus.cc:2258] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:35.682812 10509 ts_tablet_manager.cc:1905] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:35.684177 10510 ts_tablet_manager.cc:1905] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:35.687530 10510 ts_tablet_manager.cc:1918] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:35.693742 10509 ts_tablet_manager.cc:1918] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:35.693945 10510 log.cc:1196] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/bd21a706ce9f4dfe9f8a3d5eaa0952b5
I20260928 02:48:35.694393 10510 ts_tablet_manager.cc:1939] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P 7c63f45322f74a7886b44f89a2a963e9: Deleting consensus metadata
I20260928 02:48:35.694583 10509 log.cc:1196] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/bd21a706ce9f4dfe9f8a3d5eaa0952b5
I20260928 02:48:35.696931 10509 ts_tablet_manager.cc:1939] T bd21a706ce9f4dfe9f8a3d5eaa0952b5 P d766f00faac14f6f936cba75adf36351: Deleting consensus metadata
I20260928 02:48:35.698488 9968 catalog_manager.cc:4841] TS 7c63f45322f74a7886b44f89a2a963e9 (127.28.248.130:41557): tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 (table test-table [id=4dbe5de8a08243df8e0c93aceef2eb4c]) successfully deleted
I20260928 02:48:35.699869 9970 catalog_manager.cc:4841] TS d766f00faac14f6f936cba75adf36351 (127.28.248.131:35229): tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5 (table test-table [id=4dbe5de8a08243df8e0c93aceef2eb4c]) successfully deleted
W20260928 02:48:35.700560 9969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:39567 (error 108)
I20260928 02:48:35.704797 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10185
I20260928 02:48:35.713080 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10319
I20260928 02:48:35.715261 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:35.714943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39567 (local address 127.0.0.1:59200)
0928 02:48:35.715060 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:35.715149 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:48:35.716676 9969 catalog_manager.cc:4584] TS 1a42a9ae5e05427ea79dd63bc61623b0 (127.28.248.129:39567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd21a706ce9f4dfe9f8a3d5eaa0952b5: Network error: Client connection negotiation failed: client connection to 127.28.248.129:39567: connect: Connection refused (error 111)
I20260928 02:48:35.722757 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 9951
I20260928 02:48:35.730719 29666 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:48:36.076146 29666 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:48:36.076269 29666 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:48:36.421861 29666 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:48:36.422 UTC [9018] LOG: received smart shutdown request
2026-09-28 02:48:36.427 UTC [9018] LOG: background worker "logical replication launcher" (PID 9027) exited with exit code 1
2026-09-28 02:48:36.427 UTC [9022] LOG: shutting down
2026-09-28 02:48:36.594 UTC [9018] LOG: database system is shut down
2026-09-28T02:48:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47603 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:36 dist-test-slave-r238 krb5kdc[10520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:36 dist-test-slave-r238 krb5kdc[10520](info): set up 2 sockets
Sep 28 02:48:36 dist-test-slave-r238 krb5kdc[10520](info): commencing operation
krb5kdc: starting...
W20260928 02:48:37.772764 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:48:37 dist-test-slave-r238 krb5kdc[10520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:48:37.893362 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41989
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:41989,127.28.248.189:32983,127.28.248.188:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:38.001080 10533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:38.001346 10533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:38.001386 10533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:38.005330 10533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:38.005388 10533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:38.005406 10533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:38.005424 10533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:38.005445 10533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:38.010551 10533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:41989,127.28.248.189:32983,127.28.248.188:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:38.011544 10533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:38.012502 10533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:38.020915 10541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:38.020925 10539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:38.020925 10538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:38.021227 10533 server_base.cc:1034] running on GCE node
I20260928 02:48:38.021754 10533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:38.022742 10533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:38.023926 10533 hybrid_clock.cc:648] HybridClock initialized: now 1790563718023893 us; error 49 us; skew 500 ppm
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:38.026615 10533 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:38.027506 10533 webserver.cc:460] Webserver started at http://127.28.248.190:41697/ using document root <none> and password file <none>
I20260928 02:48:38.028049 10533 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:38.028134 10533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:38.028343 10533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:38.030359 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "9fe64e288a9946d5b702dcd5695678c4"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "141e09f980a0ffd1d8dd5e17b5c69b4b"
server_key_iv: "4d4b6a94c882bce493044bc4f1e0373f"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.030902 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "9fe64e288a9946d5b702dcd5695678c4"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "141e09f980a0ffd1d8dd5e17b5c69b4b"
server_key_iv: "4d4b6a94c882bce493044bc4f1e0373f"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.034597 10533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:48:38.036831 10548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:38.037968 10533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:48:38.038097 10533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "9fe64e288a9946d5b702dcd5695678c4"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "141e09f980a0ffd1d8dd5e17b5c69b4b"
server_key_iv: "4d4b6a94c882bce493044bc4f1e0373f"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.038199 10533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:38.074747 10533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:38.078212 10533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:38.078420 10533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:38.085819 10533 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:41989
I20260928 02:48:38.086107 10600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:41989 every 8 connection(s)
I20260928 02:48:38.086784 10533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:38.089252 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10533
I20260928 02:48:38.089408 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:48:38.089694 29666 external_mini_cluster.cc:1428] Setting key 3e3423d3aa8ad5fbf2f7743d9fecb161
I20260928 02:48:38.089694 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:38.093575 10601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.248.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.189@KRBTEST.COM" created.
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:38.109104 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.095950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41989 (local address 127.0.0.1:54950)
0928 02:48:38.096581 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.096654 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.097654 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.098419 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.098427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.098783 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.099035 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.099048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.100457 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.100460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.101233 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.101242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.101904 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.101908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.102015 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.102594 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.102610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.103976 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.106343 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.106350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.106358 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.106587 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.106955 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.106957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.106959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.107063 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.107783 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.107788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.107904 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.108036 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.108271 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":137,"threads_started":1}
I20260928 02:48:38.109102 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.095950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.28.248.190:41989)
0928 02:48:38.096646 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.096669 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.096774 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:48:38.097971 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.097976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.098154 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.098406 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.099168 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.100319 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.101387 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.101748 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.101920 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.104122 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.104131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.104147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.104177 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.106236 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.106694 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.106702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.106706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.106856 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.107156 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.107160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.107163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.107672 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.107796 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.108148 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.108349 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":526,"thread_start_us":204,"threads_started":1}
I20260928 02:48:38.111533 10601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.112229 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.111997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33604)
0928 02:48:38.112103 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.112172 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:48:38.112396 10551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
I20260928 02:48:38.113230 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.113018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33612)
0928 02:48:38.113118 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.113170 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:48:38.113389 10601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:32983: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab.
Entry for principal kudu/127.28.248.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab.
I20260928 02:48:38.115679 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:32983
--webserver_interface=127.28.248.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab
--principal=kudu/127.28.248.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:41989,127.28.248.189:32983,127.28.248.188:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:38.162612 10601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } attempt: 1
I20260928 02:48:38.163548 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.163224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33626)
0928 02:48:38.163383 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.163448 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:48:38.164281 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.164076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33632)
0928 02:48:38.164180 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.164229 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:48:38.164403 10601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:32983: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
W20260928 02:48:38.224488 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:38.224793 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:38.224874 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:38.228855 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:38.228930 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:38.228967 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:38.229017 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:38.229050 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:38.232549 10601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } attempt: 2
I20260928 02:48:38.233382 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.233103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33648)
0928 02:48:38.233244 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.233312 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:48:38.234031 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.233830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33658)
0928 02:48:38.233927 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.233973 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:48:38.234153 10601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:32983: Network error: Client connection negotiation failed: client connection to 127.28.248.189:32983: connect: Connection refused (error 111)
I20260928 02:48:38.234436 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:41989,127.28.248.189:32983,127.28.248.188:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab
--principal=kudu/127.28.248.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:32983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.248.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:38.235493 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:38.236515 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:48:38.244737 10608 server_base.cc:1034] running on GCE node
W20260928 02:48:38.244720 10613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:38.244683 10614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:38.244670 10616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:38.245496 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:38.246521 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:38.247699 10608 hybrid_clock.cc:648] HybridClock initialized: now 1790563718247685 us; error 27 us; skew 500 ppm
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:38.250378 10608 init.cc:377] Logged in from keytab as kudu/127.28.248.189@KRBTEST.COM (short username kudu)
I20260928 02:48:38.251250 10608 webserver.cc:460] Webserver started at http://127.28.248.189:38605/ using document root <none> and password file <none>
I20260928 02:48:38.251781 10608 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:38.251868 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:38.252080 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:38.254073 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data/instance:
uuid: "0d1b31c461244781871335971283b25b"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "31671692e050bfc8a3a6a954d5356883"
server_key_iv: "a1e21dee64ac91bdf1bb0c44fc002cef"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.254619 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal/instance:
uuid: "0d1b31c461244781871335971283b25b"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "31671692e050bfc8a3a6a954d5356883"
server_key_iv: "a1e21dee64ac91bdf1bb0c44fc002cef"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.258128 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:48:38.260407 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:38.261650 10608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:38.261781 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal
uuid: "0d1b31c461244781871335971283b25b"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "31671692e050bfc8a3a6a954d5356883"
server_key_iv: "a1e21dee64ac91bdf1bb0c44fc002cef"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.261885 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:38.306911 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:38.310117 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:38.310316 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:38.317056 10608 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.189:32983
I20260928 02:48:38.317396 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.189:32983 every 8 connection(s)
I20260928 02:48:38.318006 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
I20260928 02:48:38.320935 10676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:38.321857 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10608
I20260928 02:48:38.321954 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-1/wal/instance
I20260928 02:48:38.322244 29666 external_mini_cluster.cc:1428] Setting key 1b4d3cb8ca7a95e2898c837eff1f42a9
I20260928 02:48:38.324915 10676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.325331 10601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } attempt: 3
WARNING: no policy specified for kudu/127.28.248.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.188@KRBTEST.COM" created.
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:38.339505 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.325866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33660)
0928 02:48:38.326096 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.326113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.326213 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.328577 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.328580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.328814 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.328946 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.328954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.330659 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.330664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.331538 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.331549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.332386 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.332390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.332560 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.333200 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.333216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.335277 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.337561 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.337565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.337567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.337784 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.338128 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.338133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.338135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.338226 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.338875 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.338879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.338989 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.339123 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.339281 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":39}
I20260928 02:48:38.340431 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.326988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.28.248.190:41989)
0928 02:48:38.327116 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.327121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.327504 (+ 383us) server_negotiation.cc:407] Connection header received
0928 02:48:38.328375 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.328380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.328449 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.328550 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.329624 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.330485 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.331868 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.332244 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.332376 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.335610 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.335619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.335622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.335644 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.338275 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.338903 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.338909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.338912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.338997 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.339330 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.339334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.339337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.339589 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.339666 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.340150 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.340273 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260928 02:48:38.340780 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.326939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41989 (local address 127.0.0.1:54956)
0928 02:48:38.327365 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.327414 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.328137 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.328754 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.328760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.329204 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.329456 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.329485 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.330635 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.330640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.331764 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.331772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.332408 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.332415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.332560 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.333454 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.333470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.335464 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.338420 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.338431 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.338440 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.338778 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.339119 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.339124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.339129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.339223 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.339758 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.339765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.339897 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.340084 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.340275 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":135,"threads_started":1}
I20260928 02:48:38.340783 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.326650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.28.248.189:32983)
0928 02:48:38.327222 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.327241 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.327282 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:48:38.328137 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.328161 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.328411 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.328598 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.329120 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.330532 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.331658 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.332266 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.332432 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.335428 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.335438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.335454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.335494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.337422 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.337901 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.337907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.337914 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.338027 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.338349 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.338353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.338355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.338751 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.338909 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.339279 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.339581 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":149,"threads_started":1}
I20260928 02:48:38.342316 10676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.343130 10601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.343897 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.343523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56424)
0928 02:48:38.343737 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.343803 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 02:48:38.344574 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.344325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56436)
0928 02:48:38.344459 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.344511 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 02:48:38.344774 10601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:35515: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab.
Entry for principal kudu/127.28.248.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab.
I20260928 02:48:38.349984 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:35515
--webserver_interface=127.28.248.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab
--principal=kudu/127.28.248.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:41989,127.28.248.189:32983,127.28.248.188:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:38.350983 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.342775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33668)
0928 02:48:38.342881 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.342895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.342972 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.343194 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.343197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.343414 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.343514 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.343520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.344301 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.344304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.345041 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.345050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.345563 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.345567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.345659 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.346094 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.346104 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.347395 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.349509 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.349515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.349518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.349723 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.349981 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.349984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.349986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.350086 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.350466 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.350469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.350549 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.350655 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.350809 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:48:38.351008 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.342888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.28.248.189:32983)
0928 02:48:38.342999 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.343004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.343018 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:48:38.343054 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.343057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.343100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.343183 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.343619 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.344206 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.345148 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.345392 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.345512 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.347538 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.347542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.347544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.347588 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.349398 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.349814 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.349819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.349821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.349891 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.350194 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.350198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.350200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.350379 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.350477 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.350697 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.350883 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":27}
I20260928 02:48:38.351805 10676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.352653 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.352321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56444)
0928 02:48:38.352452 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.352547 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:48:38.353204 10624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
I20260928 02:48:38.354140 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.353928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56450)
0928 02:48:38.354032 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.354082 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:48:38.354316 10676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:35515: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
I20260928 02:48:38.375945 10601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } attempt: 1
I20260928 02:48:38.376817 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.376480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56456)
0928 02:48:38.376679 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.376739 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:48:38.377497 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.377280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56460)
0928 02:48:38.377400 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.377444 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:48:38.377650 10601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:35515: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
I20260928 02:48:38.403577 10676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } attempt: 1
I20260928 02:48:38.404501 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.404147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56470)
0928 02:48:38.404323 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.404398 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 02:48:38.405328 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.405100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56476)
0928 02:48:38.405211 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.405264 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:48:38.405527 10676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:35515: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
I20260928 02:48:38.452792 10601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } attempt: 2
I20260928 02:48:38.453617 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.453328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56490)
0928 02:48:38.453484 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.453541 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:48:38.454314 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.454134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56494)
0928 02:48:38.454224 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.454267 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:48:38.454457 10601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:35515: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
W20260928 02:48:38.460403 10682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:38.460700 10682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:38.460798 10682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:38.464792 10682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:38.464836 10682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:38.464874 10682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:38.464905 10682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:38.464938 10682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:38.470294 10682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:41989,127.28.248.189:32983,127.28.248.188:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab
--principal=kudu/127.28.248.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.248.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:38.471263 10682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:38.472239 10682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:48:38.473687 10676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } attempt: 2
I20260928 02:48:38.474406 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.474149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56504)
0928 02:48:38.474280 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.474336 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:48:38.475090 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.474872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56506)
0928 02:48:38.474999 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.475041 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 02:48:38.475212 10676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:35515: Network error: Client connection negotiation failed: client connection to 127.28.248.188:35515: connect: Connection refused (error 111)
W20260928 02:48:38.480690 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:38.480726 10689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:38.480875 10682 server_base.cc:1034] running on GCE node
W20260928 02:48:38.480691 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:38.481566 10682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:38.482621 10682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:38.483798 10682 hybrid_clock.cc:648] HybridClock initialized: now 1790563718483764 us; error 44 us; skew 500 ppm
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:38.486548 10682 init.cc:377] Logged in from keytab as kudu/127.28.248.188@KRBTEST.COM (short username kudu)
I20260928 02:48:38.487381 10682 webserver.cc:460] Webserver started at http://127.28.248.188:40623/ using document root <none> and password file <none>
I20260928 02:48:38.487926 10682 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:38.488014 10682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:38.488234 10682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:38.494726 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data/instance:
uuid: "15cdad21163943b18c43d6b4964805a4"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "e9ec8084e7f8d05856b4788fe305a52e"
server_key_iv: "6ef940075a640d3473ecf5a63a867b54"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.495379 10682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal/instance:
uuid: "15cdad21163943b18c43d6b4964805a4"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "e9ec8084e7f8d05856b4788fe305a52e"
server_key_iv: "6ef940075a640d3473ecf5a63a867b54"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.499642 10682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:48:38.502293 10698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:38.503538 10682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:48:38.503685 10682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal
uuid: "15cdad21163943b18c43d6b4964805a4"
format_stamp: "Formatted at 2026-09-28 02:48:38 on dist-test-slave-r238"
server_key: "e9ec8084e7f8d05856b4788fe305a52e"
server_key_iv: "6ef940075a640d3473ecf5a63a867b54"
server_key_version: "encryptionkey@0"
I20260928 02:48:38.503798 10682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:38.522655 10682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:38.525918 10682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:38.526118 10682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:38.532859 10682 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.188:35515
I20260928 02:48:38.533175 10750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.188:35515 every 8 connection(s)
I20260928 02:48:38.533847 10682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
I20260928 02:48:38.536499 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10682
I20260928 02:48:38.536648 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563560542449-29666-0/minicluster-data/master-2/wal/instance
I20260928 02:48:38.536736 10751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:38.536964 29666 external_mini_cluster.cc:1428] Setting key c3c6aaaecdd2fa727c9e52a5c92f8f04
I20260928 02:48:38.541213 10751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:38.551257 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.538273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.28.248.188:35515)
0928 02:48:38.538707 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.538721 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.538754 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:48:38.539437 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.539455 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.539776 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.540121 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.540959 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.542101 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.543041 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.543667 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.543837 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.546250 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.546256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.546272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.546319 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.548110 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.548542 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.548549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.548556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.548733 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.549017 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.549022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.549024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.549534 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.549735 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.549968 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.550281 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"spinlock_wait_cycles":768,"thread_start_us":133,"threads_started":1}
I20260928 02:48:38.551604 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.542925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:41989 (local address 127.0.0.1:54968)
0928 02:48:38.543310 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.543348 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.543480 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.543760 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.543765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.544109 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.544223 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.544229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.544999 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.545003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.545614 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.545622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.546150 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.546154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.546250 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.546725 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.546737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.548125 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.549985 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.549994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.550000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.550307 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.550583 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.550585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.550586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.550656 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.551055 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.551059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.551144 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.551247 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.551433 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
I20260928 02:48:38.551625 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.542982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.28.248.190:41989)
0928 02:48:38.543120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.543123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.543425 (+ 302us) server_negotiation.cc:407] Connection header received
0928 02:48:38.543578 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.543586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.543629 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.543712 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.544345 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.544886 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.545730 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.546007 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.546115 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.548313 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.548316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.548318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.548340 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.549875 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.550402 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.550406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.550407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.550499 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.550731 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.550733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.550735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.550943 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.551043 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.551352 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.551480 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:48:38.553061 10751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.553680 10601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } attempt: 3
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:38.562690 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.553621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.28.248.189:32983)
0928 02:48:38.553966 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.553972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.553985 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:48:38.554030 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.554033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.554075 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.554164 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.555001 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.555508 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.556343 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.556733 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.556806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.558910 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.558914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.558916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.558944 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.560760 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.561243 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.561247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.561249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.561328 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.561658 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.561661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.561664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.561902 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.561980 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.562365 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.562500 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260928 02:48:38.562698 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.553483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:32983 (local address 127.0.0.1:33678)
0928 02:48:38.553622 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.553635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.553697 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.554580 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.554584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.554749 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.554876 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.554883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.555615 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.555620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.556208 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.556217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.556970 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.556974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.557077 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.557520 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.557530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.558776 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.560893 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.560896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.560898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.561151 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.561436 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.561442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.561444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.561535 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.562063 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.562068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.562173 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.562280 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.562531 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:48:38.563491 10751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:38.563901 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.553990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56530)
0928 02:48:38.554162 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.554173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.554227 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.554415 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.554416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.554543 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.554613 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.554617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.556078 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.556082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.556578 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.556585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.557108 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.557111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.557294 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.557710 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.557723 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.559551 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.561840 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.561843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.561845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.562107 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.562390 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.562393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.562395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.562475 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.563438 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.563442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.563539 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.563645 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.563769 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":40}
I20260928 02:48:38.564527 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.554084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.28.248.188:35515)
0928 02:48:38.554192 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.554196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.554208 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:48:38.554293 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.554296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.554334 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.554407 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.555231 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.555956 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.556722 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.557000 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.557092 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.559960 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.559964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.559967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.559993 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.561719 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.562218 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.562221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.562222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.562293 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.562585 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.562588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.562590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.563332 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.563437 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.564221 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.564390 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:48:38.564814 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.553522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.28.248.190:41989)
0928 02:48:38.554308 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.554312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.554324 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:48:38.554356 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.554361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.554404 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.554483 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.555184 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.555931 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.556963 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.557374 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.557487 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.561197 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.561202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.561207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.561232 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.562805 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.563318 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.563321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.563323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.563458 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.563719 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.563722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.563724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.563939 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.564158 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.564515 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.564667 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":721}
I20260928 02:48:38.566349 10676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } attempt: 3
I20260928 02:48:38.568223 10601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: Bootstrap starting.
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:48:38.571473 10601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:38.572430 10601 log.cc:824] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:48:38.574132 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.563932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56534)
0928 02:48:38.564816 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.564832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.564915 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.565163 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.565166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.565334 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.565453 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.565460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.567117 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.567120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.567860 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.567870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.568293 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.568297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.568376 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.569100 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.569113 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.570460 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.572372 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.572375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.572377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.572631 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.572927 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.572929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.572931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.573011 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.573705 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.573707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.573776 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.573864 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.573999 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819}
I20260928 02:48:38.574584 10601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: No bootstrap required, opened a new log
I20260928 02:48:38.574699 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.564139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.28.248.188:35515)
0928 02:48:38.564457 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.564466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.564993 (+ 527us) server_negotiation.cc:407] Connection header received
0928 02:48:38.565021 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.565024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.565056 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.565124 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.565542 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.567025 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.567974 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.568195 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.568269 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.570588 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.570590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.570593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.570612 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.572257 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.572715 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.572721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.572726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.572818 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.573353 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.573357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.573358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.573582 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.573672 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.573958 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.574505 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260928 02:48:38.577550 10751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: Bootstrap starting.
I20260928 02:48:38.577713 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.566718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.28.248.188:35515)
0928 02:48:38.568850 (+ 2132us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.568854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.568866 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:48:38.568901 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.568903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.568933 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.569547 (+ 614us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.569624 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.570374 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.571535 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.571892 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.573553 (+ 1661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.574503 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.574506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.574509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.574530 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.576117 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.576519 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.576524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.576526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.576639 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.576894 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.576897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.576899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.577139 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.577238 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.577485 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.577578 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2063}
I20260928 02:48:38.577728 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.566685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:35515 (local address 127.0.0.1:56536)
0928 02:48:38.566791 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:38.566804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:38.566878 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:38.569077 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:38.569082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:38.569261 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:38.569383 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.569391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.570510 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.570514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.571341 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:38.571352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.572026 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.572029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:38.572127 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.572905 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:38.572921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:38.574363 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:38.576233 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.576237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.576239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.576427 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.576715 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:38.576720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:38.576722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:38.576803 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:38.577243 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:38.577247 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:38.577338 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:48:38.577438 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:48:38.577568 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":3}
I20260928 02:48:38.578575 10601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.578809 10601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:38.578860 10601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe64e288a9946d5b702dcd5695678c4, State: Initialized, Role: FOLLOWER
I20260928 02:48:38.579335 10601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.580067 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b: Bootstrap starting.
I20260928 02:48:38.580130 10751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:38.579838 10758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.580217 10758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:38.580549 10601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:38.580953 10751 log.cc:824] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:38.582434 10676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:38.583071 10751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: No bootstrap required, opened a new log
I20260928 02:48:38.583087 10676 log.cc:824] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:48:38 dist-test-slave-r238 krb5kdc[10520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
I20260928 02:48:38.585605 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b: No bootstrap required, opened a new log
I20260928 02:48:38.586920 10751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.587144 10751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:38.587193 10751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15cdad21163943b18c43d6b4964805a4, State: Initialized, Role: FOLLOWER
W20260928 02:48:38.587266 10770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:38.587320 10770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:38.587667 10751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.588990 10751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:38.589334 10676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.589524 10676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:38.589566 10676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1b31c461244781871335971283b25b, State: Initialized, Role: FOLLOWER
I20260928 02:48:38.589972 10676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.587018 10533 master_runner.cc:186] Error getting master registration for 127.28.248.189:32983: OK, instance_id { permanent_uuid: "0d1b31c461244781871335971283b25b" instance_seqno: 1790563718315215 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:48:38.590634 10533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:38.590875 10779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.591080 10779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:38.591197 10676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:38.592855 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.592976 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:48:38.595484 10785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:38.595606 10785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 02:48:38.595667 10796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:38.595746 10796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:38.597536 10682 master_runner.cc:263] No leader master found from master 15cdad21163943b18c43d6b4964805a4
I20260928 02:48:38.597608 10682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:38.599263 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.568730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.28.248.189:32983)
0928 02:48:38.568828 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.568832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.568844 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:48:38.568875 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.568878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.568913 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.568994 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.569712 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.570423 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.572025 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.572686 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.572792 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.595295 (+ 22503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.595299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.595301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.595327 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.597032 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.597532 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.597536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.597538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.597619 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.597941 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.597944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.597946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.598164 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.598334 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.599008 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.599117 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":257,"server-negotiator.queue_time_us":30}
I20260928 02:48:38.602057 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:48:38.605996 10608 master_runner.cc:263] No leader master found from master 0d1b31c461244781871335971283b25b
I20260928 02:48:38.606065 10608 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:38.625293 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.606476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.28.248.190:41989)
0928 02:48:38.606591 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.606596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.608490 (+ 1894us) server_negotiation.cc:407] Connection header received
0928 02:48:38.608695 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.608699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.608751 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.608802 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.611195 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.611998 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.615216 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.615576 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.615663 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.619077 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.619081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.619084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.619111 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.622013 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.622792 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.622796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.622798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.622877 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.623578 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.623583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.623587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.623824 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.623973 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.624954 (+ 981us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.625086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:48:38.627168 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.607154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.28.248.189:32983)
0928 02:48:38.607252 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.607256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.609972 (+ 2716us) server_negotiation.cc:407] Connection header received
0928 02:48:38.610139 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.610143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.610188 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.610240 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.613304 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.614084 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.616196 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.616538 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.616641 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.619456 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.619463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.619467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.619496 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.621348 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.623199 (+ 1851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.623203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.623205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.623279 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.624279 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.624282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.624284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.624494 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.624601 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.626890 (+ 2289us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.627020 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:48:38.628145 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:38.607861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.28.248.188:35515)
0928 02:48:38.607992 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:38.607996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:38.619753 (+ 11757us) server_negotiation.cc:407] Connection header received
0928 02:48:38.619928 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:38.619931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:38.619990 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:38.620054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:38.620877 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.621652 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.622781 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:38.623124 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:38.623192 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:38.624782 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:38.624786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:38.624789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:38.624815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:38.626403 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.626853 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.626857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.626859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.626932 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:38.627221 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:38.627224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:38.627226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:38.627433 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:38.627534 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:38.627859 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:48:38.628000 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:48:38.740785 10779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:38.740993 10779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.741940 10779 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe64e288a9946d5b702dcd5695678c4 (127.28.248.190:41989), 15cdad21163943b18c43d6b4964805a4 (127.28.248.188:35515)
I20260928 02:48:38.742616 10575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d1b31c461244781871335971283b25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe64e288a9946d5b702dcd5695678c4" is_pre_election: true
I20260928 02:48:38.742929 10575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1b31c461244781871335971283b25b in term 0.
I20260928 02:48:38.743480 10725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d1b31c461244781871335971283b25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15cdad21163943b18c43d6b4964805a4" is_pre_election: true
I20260928 02:48:38.743759 10725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1b31c461244781871335971283b25b in term 0.
I20260928 02:48:38.744194 10626 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1b31c461244781871335971283b25b, 9fe64e288a9946d5b702dcd5695678c4; no voters:
I20260928 02:48:38.744846 10779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:48:38.744939 10779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:48:38.744992 10779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:38.746352 10779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.746701 10779 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [CANDIDATE]: Term 1 election: Requested vote from peers 9fe64e288a9946d5b702dcd5695678c4 (127.28.248.190:41989), 15cdad21163943b18c43d6b4964805a4 (127.28.248.188:35515)
I20260928 02:48:38.747427 10725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d1b31c461244781871335971283b25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15cdad21163943b18c43d6b4964805a4"
I20260928 02:48:38.747565 10725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:38.748741 10575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d1b31c461244781871335971283b25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe64e288a9946d5b702dcd5695678c4"
I20260928 02:48:38.748875 10575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:38.750128 10575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1b31c461244781871335971283b25b in term 1.
I20260928 02:48:38.750458 10626 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1b31c461244781871335971283b25b, 9fe64e288a9946d5b702dcd5695678c4; no voters:
I20260928 02:48:38.750972 10725 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1b31c461244781871335971283b25b in term 1.
I20260928 02:48:38.751441 10779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:38.751686 10779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [term 1 LEADER]: Becoming Leader. State: Replica: 0d1b31c461244781871335971283b25b, State: Running, Role: LEADER
I20260928 02:48:38.752027 10779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } }
I20260928 02:48:38.753906 10779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1b31c461244781871335971283b25b. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.754024 10779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:38.756996 10808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:38.758114 10808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:38.764811 10725 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [term 1 FOLLOWER]: Refusing update from remote peer 0d1b31c461244781871335971283b25b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:38.765574 10779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:38.766120 10575 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [term 1 FOLLOWER]: Refusing update from remote peer 0d1b31c461244781871335971283b25b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:38.766793 10806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:38.770082 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1b31c461244781871335971283b25b. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.770221 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:38.770704 10758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1b31c461244781871335971283b25b. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.770828 10758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:38.775391 10779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.775516 10779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:38.775774 10758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.775875 10758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:38.776165 10809 mvcc.cc:205] Tried to move back new op lower bound from 7334148992053530624 to 7334148992011669504. Current Snapshot: MvccSnapshot[applied={T|T < 7334148992053530624}]
W20260928 02:48:38.777423 10704 tablet.cc:2369] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:48:38.777945 10811 mvcc.cc:205] Tried to move back new op lower bound from 7334148992053530624 to 7334148992011669504. Current Snapshot: MvccSnapshot[applied={T|T < 7334148992053530624}]
I20260928 02:48:38.779114 10806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.779237 10806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:38.780067 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d1b31c461244781871335971283b25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe64e288a9946d5b702dcd5695678c4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 41989 } } peers { permanent_uuid: "0d1b31c461244781871335971283b25b" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 32983 } } peers { permanent_uuid: "15cdad21163943b18c43d6b4964805a4" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 35515 } } }
I20260928 02:48:38.780179 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:38.780827 10810 mvcc.cc:205] Tried to move back new op lower bound from 7334148992053530624 to 7334148992011669504. Current Snapshot: MvccSnapshot[applied={T|T < 7334148992053530624}]
I20260928 02:48:38.782539 10808 catalog_manager.cc:1347] Generated new cluster ID: 5c4be28773484b929edb16c536ec943e
I20260928 02:48:38.782596 10808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:38.803499 10808 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:38.804479 10808 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:38.811553 10808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d1b31c461244781871335971283b25b: Generated new TSK 0
I20260928 02:48:38.812363 10808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:48:39.588646 10770 catalog_manager.cc:1259] Loaded cluster ID: 5c4be28773484b929edb16c536ec943e
I20260928 02:48:39.588732 10770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: loading cluster ID for follower catalog manager: success
I20260928 02:48:39.590517 10770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: acquiring CA information for follower catalog manager: success
I20260928 02:48:39.591738 10770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fe64e288a9946d5b702dcd5695678c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:48:39.596715 10785 catalog_manager.cc:1259] Loaded cluster ID: 5c4be28773484b929edb16c536ec943e
I20260928 02:48:39.596771 10785 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: loading cluster ID for follower catalog manager: success
I20260928 02:48:39.598773 10785 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: acquiring CA information for follower catalog manager: success
I20260928 02:48:39.599840 10785 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15cdad21163943b18c43d6b4964805a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:48:39.865214 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10533
I20260928 02:48:39.871335 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10608
I20260928 02:48:39.877048 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10682
2026-09-28T02:48:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3159 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:39 dist-test-slave-r238 krb5kdc[10823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:39 dist-test-slave-r238 krb5kdc[10823](info): set up 2 sockets
Sep 28 02:48:39 dist-test-slave-r238 krb5kdc[10823](info): commencing operation
krb5kdc: starting...
W20260928 02:48:40.936874 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:48:41 dist-test-slave-r238 krb5kdc[10823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:48:41.060484 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36241
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:41.172066 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:41.172333 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:41.172374 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:41.176499 10836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:41.176558 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:41.176578 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:41.176636 10836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:41.176668 10836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:41.182471 10836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:41.183545 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:41.184587 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:41.193011 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:41.193050 10836 server_base.cc:1034] running on GCE node
W20260928 02:48:41.193001 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:41.192999 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:41.193764 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:41.194826 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:41.196099 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790563721196084 us; error 28 us; skew 500 ppm
Sep 28 02:48:41 dist-test-slave-r238 krb5kdc[10823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563721, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:41.198858 10836 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:41.199743 10836 webserver.cc:460] Webserver started at http://127.28.248.190:46343/ using document root <none> and password file <none>
I20260928 02:48:41.200292 10836 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:41.200377 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:41.200594 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:41.202610 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "77e26a5b87ca4da4999c7ed04c31059d"
format_stamp: "Formatted at 2026-09-28 02:48:41 on dist-test-slave-r238"
server_key: "e8e85e0be14f16795a78f9fc17bb2371"
server_key_iv: "75ed6370be3480d83d78a3d8688537f5"
server_key_version: "encryptionkey@0"
I20260928 02:48:41.203159 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "77e26a5b87ca4da4999c7ed04c31059d"
format_stamp: "Formatted at 2026-09-28 02:48:41 on dist-test-slave-r238"
server_key: "e8e85e0be14f16795a78f9fc17bb2371"
server_key_iv: "75ed6370be3480d83d78a3d8688537f5"
server_key_version: "encryptionkey@0"
I20260928 02:48:41.206733 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 02:48:41.209065 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:41.210227 10836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:48:41.210356 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "77e26a5b87ca4da4999c7ed04c31059d"
format_stamp: "Formatted at 2026-09-28 02:48:41 on dist-test-slave-r238"
server_key: "e8e85e0be14f16795a78f9fc17bb2371"
server_key_iv: "75ed6370be3480d83d78a3d8688537f5"
server_key_version: "encryptionkey@0"
I20260928 02:48:41.210461 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:41.237920 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:41.240835 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:41.241048 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:41.248062 10836 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:36241
I20260928 02:48:41.248085 10905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:36241 every 8 connection(s)
I20260928 02:48:41.249046 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:41.251734 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:41.256918 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10836
I20260928 02:48:41.257046 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:48:41.257331 29666 external_mini_cluster.cc:1428] Setting key c2c27421cb653c537052d3d63d91095b
I20260928 02:48:41.257403 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d: Bootstrap starting.
I20260928 02:48:41.259806 10906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:41.260506 10906 log.cc:824] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:41.262275 10906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d: No bootstrap required, opened a new log
Sep 28 02:48:41 dist-test-slave-r238 krb5kdc[10823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:41.265892 10906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e26a5b87ca4da4999c7ed04c31059d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36241 } }
I20260928 02:48:41.266186 10906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:41.266249 10906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e26a5b87ca4da4999c7ed04c31059d, State: Initialized, Role: FOLLOWER
I20260928 02:48:41.266747 10906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e26a5b87ca4da4999c7ed04c31059d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36241 } }
I20260928 02:48:41.266908 10906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:41.266994 10906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:41.267103 10906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:41.268270 10906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e26a5b87ca4da4999c7ed04c31059d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36241 } }
I20260928 02:48:41.268656 10906 leader_election.cc:302] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77e26a5b87ca4da4999c7ed04c31059d; no voters:
I20260928 02:48:41.269021 10906 leader_election.cc:288] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:48:41.269121 10911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:41.269325 10911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [term 1 LEADER]: Becoming Leader. State: Replica: 77e26a5b87ca4da4999c7ed04c31059d, State: Running, Role: LEADER
I20260928 02:48:41.269655 10911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e26a5b87ca4da4999c7ed04c31059d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36241 } }
I20260928 02:48:41.270208 10906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:41.272176 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:41.258596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.28.248.190:36241)
0928 02:48:41.259015 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:41.259030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:41.259064 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:48:41.259671 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:41.259687 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:41.260054 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:41.260370 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:41.261180 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:41.262141 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:41.263025 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:41.263477 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:41.263651 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:41.266397 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:41.266403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:41.266419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:41.266459 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:41.268493 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:41.268980 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:41.268987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:41.268993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:41.269134 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:41.269615 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:41.269619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:41.269621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:41.270138 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:41.270320 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:41.270770 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 02:48:41.271109 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20260928 02:48:41.273085 10912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77e26a5b87ca4da4999c7ed04c31059d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e26a5b87ca4da4999c7ed04c31059d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36241 } } }
I20260928 02:48:41.273203 10912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:41.273488 10913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77e26a5b87ca4da4999c7ed04c31059d. Latest consensus state: current_term: 1 leader_uuid: "77e26a5b87ca4da4999c7ed04c31059d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e26a5b87ca4da4999c7ed04c31059d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36241 } } }
I20260928 02:48:41.273586 10913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:41.275306 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 02:48:41.275743 10925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:41.275816 10925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:41.275902 10927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:41.276777 10927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:41.281791 10927 catalog_manager.cc:1347] Generated new cluster ID: 673a3c1fa730438e8823694eaf2369c3
I20260928 02:48:41.281858 10927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:41.292932 10927 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:41.294235 10927 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:41.300534 10927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77e26a5b87ca4da4999c7ed04c31059d: Generated new TSK 0
I20260928 02:48:41.301347 10927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:48:41.331295 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10836
2026-09-28T02:48:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1451 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:41 dist-test-slave-r238 krb5kdc[10933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:41 dist-test-slave-r238 krb5kdc[10933](info): set up 2 sockets
Sep 28 02:48:41 dist-test-slave-r238 krb5kdc[10933](info): commencing operation
krb5kdc: starting...
W20260928 02:48:42.465088 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:48:42 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:48:42.585327 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45511
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:42.693359 10946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:42.693652 10946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:42.693742 10946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:42.697736 10946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:42.697799 10946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:42.697819 10946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:42.697835 10946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:42.697850 10946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:42.703127 10946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:45511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:42.704190 10946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:42.705627 10946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:42.713905 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:42.713905 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:42.714001 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:42.714172 10946 server_base.cc:1034] running on GCE node
I20260928 02:48:42.714735 10946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:42.715771 10946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:42.716928 10946 hybrid_clock.cc:648] HybridClock initialized: now 1790563722716895 us; error 46 us; skew 500 ppm
Sep 28 02:48:42 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:42.719934 10946 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:42.721002 10946 webserver.cc:460] Webserver started at http://127.28.248.190:36855/ using document root <none> and password file <none>
I20260928 02:48:42.721541 10946 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:42.721601 10946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:42.721767 10946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:42.723742 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "2782de666b8b44798b21906ab2e45f23"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "0a6cf6d32c8b620d775c9a1174eb4586"
server_key_iv: "4673c2eb51fa5b486854667654e842d1"
server_key_version: "encryptionkey@0"
I20260928 02:48:42.724262 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "2782de666b8b44798b21906ab2e45f23"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "0a6cf6d32c8b620d775c9a1174eb4586"
server_key_iv: "4673c2eb51fa5b486854667654e842d1"
server_key_version: "encryptionkey@0"
I20260928 02:48:42.727903 10946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:48:42.730279 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:42.731787 10946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:48:42.731951 10946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "2782de666b8b44798b21906ab2e45f23"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "0a6cf6d32c8b620d775c9a1174eb4586"
server_key_iv: "4673c2eb51fa5b486854667654e842d1"
server_key_version: "encryptionkey@0"
I20260928 02:48:42.732065 10946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:42.746901 10946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:42.749864 10946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:42.750022 10946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:42.757547 10946 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:45511
I20260928 02:48:42.757812 11013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:45511 every 8 connection(s)
I20260928 02:48:42.758600 10946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:42.760987 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 10946
I20260928 02:48:42.761129 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:48:42.761394 29666 external_mini_cluster.cc:1428] Setting key 2046dcf906a148275d76b03b5ec16fac
I20260928 02:48:42.761579 11014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:42.768760 11014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap starting.
Sep 28 02:48:42 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:42.771019 11014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:42.771916 11014 log.cc:824] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:42.774179 11014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: No bootstrap required, opened a new log
I20260928 02:48:42.775969 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:42.762820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.28.248.190:45511)
0928 02:48:42.763232 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:42.763251 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:42.763292 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:48:42.764258 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:42.764284 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:42.764747 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:42.765125 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:42.765788 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:42.767229 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:42.768070 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:42.768787 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:42.768951 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:42.771382 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:42.771388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:42.771405 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:42.771435 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:42.773219 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:42.773707 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:42.773711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:42.773715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:42.773816 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:42.774107 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:42.774109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:42.774110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:42.774520 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:42.774647 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:42.774954 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 02:48:42.775251 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260928 02:48:42.778148 11014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:42.778378 11014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:42.778458 11014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2782de666b8b44798b21906ab2e45f23, State: Initialized, Role: FOLLOWER
I20260928 02:48:42.778856 11014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:42.779002 11014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:42.779080 11014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:42.779191 11014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:42.780139 11014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:42.780465 11014 leader_election.cc:302] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2782de666b8b44798b21906ab2e45f23; no voters:
I20260928 02:48:42.780799 11014 leader_election.cc:288] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:48:42.780884 11019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:42.781114 11019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 LEADER]: Becoming Leader. State: Replica: 2782de666b8b44798b21906ab2e45f23, State: Running, Role: LEADER
I20260928 02:48:42.781395 11019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:42.782091 11014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:42.783067 11020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2782de666b8b44798b21906ab2e45f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } } }
I20260928 02:48:42.783205 11020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:42.783567 11028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:42.783541 11021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2782de666b8b44798b21906ab2e45f23. Latest consensus state: current_term: 1 leader_uuid: "2782de666b8b44798b21906ab2e45f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } } }
I20260928 02:48:42.784147 11021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:42.786185 11028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:42.792104 11028 catalog_manager.cc:1347] Generated new cluster ID: 473252ef229d4468aedab94eba325db6
I20260928 02:48:42.792183 11028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:42.799285 11028 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:42.800177 11028 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:42.807509 11028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Generated new TSK 0
I20260928 02:48:42.808137 11028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:48:42.843679 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45511
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:42.950549 11040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:42.950836 11040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:42.950901 11040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:42.954901 11040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:42.954962 11040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:42.955060 11040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:48:42.960067 11040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:42.961087 11040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:42.962054 11040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:42.971367 11046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:42.971391 11045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:42.971391 11048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:42.971588 11040 server_base.cc:1034] running on GCE node
I20260928 02:48:42.971941 11040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:42.972518 11040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:42.973695 11040 hybrid_clock.cc:648] HybridClock initialized: now 1790563722973679 us; error 26 us; skew 500 ppm
Sep 28 02:48:42 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:42.976220 11040 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:48:42.977240 11040 webserver.cc:460] Webserver started at http://127.28.248.129:41177/ using document root <none> and password file <none>
I20260928 02:48:42.977784 11040 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:42.977869 11040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:42.978078 11040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:42.980089 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "718927e002354f308fcd86723c8e9e48"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "7731396fae45839f3162e18a11fc35a8"
server_key_iv: "2042ca5196566c5aa6a7f95c28c9d2bc"
server_key_version: "encryptionkey@0"
I20260928 02:48:42.980706 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "718927e002354f308fcd86723c8e9e48"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "7731396fae45839f3162e18a11fc35a8"
server_key_iv: "2042ca5196566c5aa6a7f95c28c9d2bc"
server_key_version: "encryptionkey@0"
I20260928 02:48:42.984413 11040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:48:42.986763 11055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:42.987916 11040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:42.988039 11040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "718927e002354f308fcd86723c8e9e48"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "7731396fae45839f3162e18a11fc35a8"
server_key_iv: "2042ca5196566c5aa6a7f95c28c9d2bc"
server_key_version: "encryptionkey@0"
I20260928 02:48:42.988148 11040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:43.004935 11040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:43.007813 11040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:43.008004 11040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:43.008560 11040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:43.009352 11040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:43.009418 11040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.009487 11040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:43.009534 11040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.017910 11040 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:39493
I20260928 02:48:43.017938 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:39493 every 8 connection(s)
I20260928 02:48:43.018728 11040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:48:43.019618 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11040
I20260928 02:48:43.019742 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:48:43.020049 29666 external_mini_cluster.cc:1428] Setting key 5d1b1345846fa9b51b48cba03bd61f82
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:48:43.035274 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.020650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:36395 (local address 127.28.248.190:45511)
0928 02:48:43.020797 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.020802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.021744 (+ 942us) server_negotiation.cc:407] Connection header received
0928 02:48:43.023757 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.023761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.023812 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.023894 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:43.025291 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.025858 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.027580 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.028018 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.028140 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.031412 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:43.031416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:43.031421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:43.031480 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:43.033224 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.033741 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.033745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.033747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.033829 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.034165 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.034168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.034170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.034397 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:43.034498 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.034970 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.035097 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:48:43.036057 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.021056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:36395)
0928 02:48:43.021571 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.021636 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.023499 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.024193 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.024201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.024680 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:43.025149 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.025163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.026007 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.026011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.027406 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.027418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.028229 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.028234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.028451 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.029165 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:43.029184 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:43.031233 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:43.033332 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:43.033336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:43.033347 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:43.033629 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:43.033924 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:43.033927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:43.033929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:43.034069 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:43.034495 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:43.034501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:43.034741 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.034914 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.035240 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":150,"threads_started":1}
I20260928 02:48:43.037458 11170 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45511
I20260928 02:48:43.037693 11170 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:43.038156 11170 heartbeater.cc:511] Master 127.28.248.190:45511 requested a full tablet report, sending...
I20260928 02:48:43.039577 10978 ts_manager.cc:194] Registered new tserver with Master: 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493)
I20260928 02:48:43.040719 10978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:36395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:48:43.051066 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45511
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:43.158617 11175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:43.158854 11175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:43.158895 11175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:43.162842 11175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:43.162899 11175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:43.162961 11175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:48:43.168056 11175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:43.169036 11175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:43.170215 11175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:48:43.180891 11175 server_base.cc:1034] running on GCE node
W20260928 02:48:43.180863 11180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.180855 11183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.180908 11181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:43.181494 11175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:43.182181 11175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:43.183339 11175 hybrid_clock.cc:648] HybridClock initialized: now 1790563723183320 us; error 35 us; skew 500 ppm
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:43.186204 11175 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:48:43.186956 11175 webserver.cc:460] Webserver started at http://127.28.248.130:40511/ using document root <none> and password file <none>
I20260928 02:48:43.187441 11175 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:43.187495 11175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:43.187664 11175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:43.189612 11175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "c3a6b366df78430c8f7b9dd2b0842d8a"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "9f11c6b0304ef59f86564e33fcbbe506"
server_key_iv: "8cc60d76f49ee216f681ab478ab4b2a1"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.190085 11175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "c3a6b366df78430c8f7b9dd2b0842d8a"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "9f11c6b0304ef59f86564e33fcbbe506"
server_key_iv: "8cc60d76f49ee216f681ab478ab4b2a1"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.193676 11175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:48:43.195777 11190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.197028 11175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:43.197140 11175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "c3a6b366df78430c8f7b9dd2b0842d8a"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "9f11c6b0304ef59f86564e33fcbbe506"
server_key_iv: "8cc60d76f49ee216f681ab478ab4b2a1"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.197233 11175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:43.206734 11175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:43.209352 11175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:43.209532 11175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:43.210034 11175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:43.210772 11175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:43.210814 11175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.210875 11175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:43.210909 11175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.218981 11175 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:41951
I20260928 02:48:43.219038 11303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:41951 every 8 connection(s)
I20260928 02:48:43.219831 11175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:48:43.226929 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11175
I20260928 02:48:43.227116 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:48:43.227421 29666 external_mini_cluster.cc:1428] Setting key b53bec9a1a64dfb5ac7c6419d691cf2c
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:43.233522 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.221570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:57781 (local address 127.28.248.190:45511)
0928 02:48:43.221705 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.221709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.222466 (+ 757us) server_negotiation.cc:407] Connection header received
0928 02:48:43.223774 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.223778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.223838 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.223939 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:43.225102 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.225644 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.226742 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.226976 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.227104 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.229855 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:43.229859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:43.229861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:43.229883 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:43.231359 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.231910 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.231915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.231917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.231998 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.232342 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.232345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.232347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.232591 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:43.232711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.233214 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.233345 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:48:43.234292 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.221899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:57781)
0928 02:48:43.222330 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.222370 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.223573 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.224116 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.224122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.224457 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:43.224957 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.224970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.225764 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.225767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.226624 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.226632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.227170 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.227176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.227375 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.228052 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:43.228072 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:43.229690 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:43.231533 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:43.231539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:43.231553 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:43.231807 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:43.232098 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:43.232101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:43.232103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:43.232245 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:43.232720 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:43.232728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:43.232974 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.233159 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.233481 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260928 02:48:43.235390 11304 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45511
I20260928 02:48:43.235659 11304 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:43.236294 11304 heartbeater.cc:511] Master 127.28.248.190:45511 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:48:43.237390 10978 ts_manager.cc:194] Registered new tserver with Master: c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
I20260928 02:48:43.237900 10978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:57781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:48:43.253750 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45511
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:43.361300 11309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:43.361582 11309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:43.361655 11309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:43.365633 11309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:43.365720 11309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:43.365806 11309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:48:43.371235 11309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:43.372300 11309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:43.373361 11309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:43.382961 11315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.382969 11317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.382997 11314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:43.383317 11309 server_base.cc:1034] running on GCE node
I20260928 02:48:43.383807 11309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:43.384491 11309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:43.385664 11309 hybrid_clock.cc:648] HybridClock initialized: now 1790563723385651 us; error 31 us; skew 500 ppm
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:43.388594 11309 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:48:43.389700 11309 webserver.cc:460] Webserver started at http://127.28.248.131:42531/ using document root <none> and password file <none>
I20260928 02:48:43.390465 11309 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:43.390554 11309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:43.390806 11309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:43.392915 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "8450c10a082d680738372108517c5a5a"
server_key_iv: "744225f6f01da6a41669e7e536852981"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.393411 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "8450c10a082d680738372108517c5a5a"
server_key_iv: "744225f6f01da6a41669e7e536852981"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.396883 11309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:48:43.399060 11324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.400195 11309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:43.400316 11309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "8450c10a082d680738372108517c5a5a"
server_key_iv: "744225f6f01da6a41669e7e536852981"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.400417 11309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:43.417973 11309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:43.421207 11309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:43.421401 11309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:43.421962 11309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:43.422734 11309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:43.422801 11309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.422869 11309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:43.422909 11309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.431794 11309 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:36985
I20260928 02:48:43.431814 11437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:36985 every 8 connection(s)
I20260928 02:48:43.432698 11309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:48:43.440176 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11309
I20260928 02:48:43.440312 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:48:43.440575 29666 external_mini_cluster.cc:1428] Setting key ae7aeb202207422d121d0b227b567070
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:43.446022 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.434391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:44769 (local address 127.28.248.190:45511)
0928 02:48:43.434524 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.434528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.435394 (+ 866us) server_negotiation.cc:407] Connection header received
0928 02:48:43.436943 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.436946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.437006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.437109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:43.438230 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.438773 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.439918 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.440165 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.440274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.442878 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:43.442881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:43.442883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:43.442906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:43.444072 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.444557 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.444562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.444564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.444687 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.445017 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.445021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.445023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.445195 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:43.445298 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.445751 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.445866 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:48:43.446772 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.434780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:44769)
0928 02:48:43.435251 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.435300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.436745 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.437275 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.437282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.437631 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:43.438085 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.438098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.438892 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.438896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.439791 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.439800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.440311 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.440315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.440479 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.441090 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:43.441105 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:43.442727 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:43.444180 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:43.444186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:43.444199 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:43.444457 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:43.444776 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:43.444780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:43.444782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:43.444918 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:43.445295 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:43.445302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:43.445533 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.445712 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.446020 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260928 02:48:43.447751 11438 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45511
I20260928 02:48:43.447945 11438 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:43.448351 11438 heartbeater.cc:511] Master 127.28.248.190:45511 requested a full tablet report, sending...
I20260928 02:48:43.449277 10978 ts_manager.cc:194] Registered new tserver with Master: 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985)
I20260928 02:48:43.449816 10978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:44769
I20260928 02:48:43.454144 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:48:43.462515 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.455963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.28.248.190:45511)
0928 02:48:43.456070 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.456073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.456297 (+ 224us) server_negotiation.cc:407] Connection header received
0928 02:48:43.456420 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.456422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.456458 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.456517 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:43.457184 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.457687 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.458443 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.458643 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.458712 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.459915 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:43.459918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:43.459919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:43.459939 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:43.461191 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.461559 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.461562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.461563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.461629 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:43.461834 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:43.461836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:43.461837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:43.462046 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:43.462141 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.462316 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.462408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:48:43.465211 10978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:48:43.468137 10978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:48:43.481912 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.475989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:41951 (local address 127.0.0.1:35478)
0928 02:48:43.477357 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.477369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.477437 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.478069 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.478075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.478095 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:43.478171 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.478175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.479164 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.479168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.480339 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.480347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.481291 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.481295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.481486 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.481526 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.481631 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.481747 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":45,"thread_start_us":67,"threads_started":1}
I20260928 02:48:43.482035 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.476149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39493 (local address 127.0.0.1:38442)
0928 02:48:43.477025 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.477074 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.477214 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.477825 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.477833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.477861 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:43.478023 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.478032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.479639 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.479642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.480507 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.480516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.481599 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.481606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.481743 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.481763 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.481828 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.481878 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":46,"spinlock_wait_cycles":712064,"thread_start_us":107,"threads_started":1}
I20260928 02:48:43.482483 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.476115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.28.248.130:41951)
0928 02:48:43.477494 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.477507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.477528 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:48:43.477599 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.477603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.477744 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.477902 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:43.478279 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.479032 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.480463 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.481156 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.481294 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.481455 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.481763 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.481883 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":148,"threads_started":1}
I20260928 02:48:43.482431 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.476494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.28.248.129:39493)
0928 02:48:43.476857 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.476886 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.477358 (+ 472us) server_negotiation.cc:407] Connection header received
0928 02:48:43.477448 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.477454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.477664 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.477829 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:43.478289 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.479482 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.480647 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.481476 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.481591 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.481711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.482161 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.482316 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260928 02:48:43.483518 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.476675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:36985 (local address 127.0.0.1:51446)
0928 02:48:43.477104 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.477117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.477214 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.477873 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.477879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.477900 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:43.478219 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.478226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.479639 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.479642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.481775 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.481784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.483071 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.483075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.483161 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.483186 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.483315 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.483379 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":377,"spinlock_wait_cycles":2816,"thread_start_us":86,"threads_started":1}
I20260928 02:48:43.483939 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.477068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.28.248.131:36985)
0928 02:48:43.477493 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.477505 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.477523 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:48:43.477585 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.477589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.477729 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.477962 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:43.478356 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.479501 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.481902 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.482915 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.483084 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.483196 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.483697 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.483799 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":100,"threads_started":1}
I20260928 02:48:43.485476 11238 tablet_service.cc:1467] Processing CreateTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e (DEFAULT_TABLE table=test-table [id=b44862b4a1304f069b0b0a84c4ace08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:43.485764 11104 tablet_service.cc:1467] Processing CreateTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e (DEFAULT_TABLE table=test-table [id=b44862b4a1304f069b0b0a84c4ace08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:43.486527 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b69766992f41c5a3228cd3bf0e8d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:43.486403 11372 tablet_service.cc:1467] Processing CreateTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e (DEFAULT_TABLE table=test-table [id=b44862b4a1304f069b0b0a84c4ace08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:43.486799 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b69766992f41c5a3228cd3bf0e8d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:43.487316 11372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b69766992f41c5a3228cd3bf0e8d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:43.492838 11455 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Bootstrap starting.
I20260928 02:48:43.494345 11456 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Bootstrap starting.
I20260928 02:48:43.495261 11455 tablet_bootstrap.cc:654] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:43.495985 11455 log.cc:824] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:43.496001 11456 tablet_bootstrap.cc:654] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:43.496924 11457 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Bootstrap starting.
I20260928 02:48:43.497167 11456 log.cc:824] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:43.498359 11455 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: No bootstrap required, opened a new log
I20260928 02:48:43.498553 11455 ts_tablet_manager.cc:1397] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:48:43.498950 11457 tablet_bootstrap.cc:654] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:43.499051 11456 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: No bootstrap required, opened a new log
I20260928 02:48:43.499239 11456 ts_tablet_manager.cc:1397] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:48:43.500170 11457 log.cc:824] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:43.501945 11457 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: No bootstrap required, opened a new log
I20260928 02:48:43.501823 11455 raft_consensus.cc:348] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.502040 11455 raft_consensus.cc:374] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:43.502092 11455 raft_consensus.cc:729] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a6b366df78430c8f7b9dd2b0842d8a, State: Initialized, Role: FOLLOWER
I20260928 02:48:43.502116 11457 ts_tablet_manager.cc:1397] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:48:43.502519 11455 consensus_queue.cc:260] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.503132 11304 heartbeater.cc:503] Master 127.28.248.190:45511 was elected leader, sending a full tablet report...
I20260928 02:48:43.503245 11456 raft_consensus.cc:348] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.503468 11456 raft_consensus.cc:374] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:43.503463 11455 ts_tablet_manager.cc:1428] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:48:43.503512 11456 raft_consensus.cc:729] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718927e002354f308fcd86723c8e9e48, State: Initialized, Role: FOLLOWER
I20260928 02:48:43.504311 11456 consensus_queue.cc:260] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.504937 11170 heartbeater.cc:503] Master 127.28.248.190:45511 was elected leader, sending a full tablet report...
I20260928 02:48:43.506138 11456 ts_tablet_manager.cc:1428] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:48:43.505980 11457 raft_consensus.cc:348] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.506208 11457 raft_consensus.cc:374] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:43.506260 11457 raft_consensus.cc:729] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f735704c39f4cb5ab2d8313890ea9d1, State: Initialized, Role: FOLLOWER
I20260928 02:48:43.506652 11457 consensus_queue.cc:260] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.507262 11438 heartbeater.cc:503] Master 127.28.248.190:45511 was elected leader, sending a full tablet report...
I20260928 02:48:43.507496 11457 ts_tablet_manager.cc:1428] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 02:48:43.519872 11171 tablet.cc:2369] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:48:43.547720 11462 raft_consensus.cc:482] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:43.547911 11462 raft_consensus.cc:504] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.549095 11462 leader_election.cc:288] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951), 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985)
I20260928 02:48:43.553927 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.549428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:52081 (local address 127.28.248.130:41951)
0928 02:48:43.549587 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.549592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.549709 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:48:43.549872 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.549875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.549917 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.549997 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:43.550279 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.550850 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.552495 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.553191 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.553347 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.553403 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.553706 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.553784 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:48:43.553957 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.549469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:54707 (local address 127.28.248.131:36985)
0928 02:48:43.549601 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.549607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.549790 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:48:43.549937 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.549940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.549980 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.550070 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:43.550337 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.550992 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.552487 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.553193 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.553347 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.553403 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.553706 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.553784 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:48:43.553969 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.549340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:36985 (local address 127.28.248.129:54707)
0928 02:48:43.549706 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.549716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.549826 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.550079 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.550082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.550097 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:43.550233 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.550238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.551106 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.551109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.552339 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.552349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.553336 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.553340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.553551 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.553595 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.553673 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.553790 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":157,"thread_start_us":117,"threads_started":1}
I20260928 02:48:43.553972 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.549298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:41951 (local address 127.28.248.129:52081)
0928 02:48:43.549611 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.549627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.549760 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.550012 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.550014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.550029 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:43.550156 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.550161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.550947 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.550953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.552339 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.552349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.553336 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.553340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:43.553546 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.553593 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:48:43.553673 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:48:43.553785 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":69,"thread_start_us":73,"threads_started":1}
I20260928 02:48:43.554636 11258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b69766992f41c5a3228cd3bf0e8d0e" candidate_uuid: "718927e002354f308fcd86723c8e9e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" is_pre_election: true
I20260928 02:48:43.554764 11392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b69766992f41c5a3228cd3bf0e8d0e" candidate_uuid: "718927e002354f308fcd86723c8e9e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" is_pre_election: true
I20260928 02:48:43.555015 11258 raft_consensus.cc:2454] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718927e002354f308fcd86723c8e9e48 in term 0.
I20260928 02:48:43.555114 11392 raft_consensus.cc:2454] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718927e002354f308fcd86723c8e9e48 in term 0.
I20260928 02:48:43.555533 11056 leader_election.cc:302] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718927e002354f308fcd86723c8e9e48, c3a6b366df78430c8f7b9dd2b0842d8a; no voters:
I20260928 02:48:43.555842 11462 raft_consensus.cc:2781] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:48:43.555926 11462 raft_consensus.cc:482] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:48:43.555968 11462 raft_consensus.cc:3037] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:43.557085 11462 raft_consensus.cc:504] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.557421 11462 leader_election.cc:288] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 1 election: Requested vote from peers c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951), 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985)
I20260928 02:48:43.557719 11258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b69766992f41c5a3228cd3bf0e8d0e" candidate_uuid: "718927e002354f308fcd86723c8e9e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a"
I20260928 02:48:43.557762 11392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b69766992f41c5a3228cd3bf0e8d0e" candidate_uuid: "718927e002354f308fcd86723c8e9e48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
I20260928 02:48:43.557850 11258 raft_consensus.cc:3037] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:43.557888 11392 raft_consensus.cc:3037] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:43.559126 11258 raft_consensus.cc:2454] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718927e002354f308fcd86723c8e9e48 in term 1.
I20260928 02:48:43.559126 11392 raft_consensus.cc:2454] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718927e002354f308fcd86723c8e9e48 in term 1.
I20260928 02:48:43.559394 11056 leader_election.cc:302] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718927e002354f308fcd86723c8e9e48, c3a6b366df78430c8f7b9dd2b0842d8a; no voters:
I20260928 02:48:43.559568 11462 raft_consensus.cc:2781] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:43.559784 11462 raft_consensus.cc:686] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 LEADER]: Becoming Leader. State: Replica: 718927e002354f308fcd86723c8e9e48, State: Running, Role: LEADER
I20260928 02:48:43.560129 11462 consensus_queue.cc:237] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:43.563159 10977 catalog_manager.cc:5495] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718927e002354f308fcd86723c8e9e48 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "718927e002354f308fcd86723c8e9e48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } health_report { overall_health: UNKNOWN } } }
I20260928 02:48:43.577849 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.574682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.28.248.129:39493)
0928 02:48:43.574787 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.574791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.574804 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:48:43.574909 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.574911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.574961 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.575047 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:43.575244 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.575730 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.576499 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:43.576766 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:43.576841 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:43.577089 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:43.577486 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:43.577584 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:43.577634 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:48:43.577679 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:48:43.583810 11258 raft_consensus.cc:1261] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Refusing update from remote peer 718927e002354f308fcd86723c8e9e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:43.583810 11392 raft_consensus.cc:1261] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Refusing update from remote peer 718927e002354f308fcd86723c8e9e48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:43.584558 11462 consensus_queue.cc:1035] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:43.584793 11466 consensus_queue.cc:1035] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:43.592034 11469 mvcc.cc:205] Tried to move back new op lower bound from 7334149011792293888 to 7334149011705008128. Current Snapshot: MvccSnapshot[applied={T|T < 7334149011792293888}]
I20260928 02:48:43.592840 11470 mvcc.cc:205] Tried to move back new op lower bound from 7334149011792293888 to 7334149011705008128. Current Snapshot: MvccSnapshot[applied={T|T < 7334149011792293888}]
I20260928 02:48:43.593391 11471 mvcc.cc:205] Tried to move back new op lower bound from 7334149011792293888 to 7334149011705008128. Current Snapshot: MvccSnapshot[applied={T|T < 7334149011792293888}]
I20260928 02:48:43.601435 10977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:48:43.601601 10977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:48:43.603889 10977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Sending DeleteTablet for 3 replicas of tablet 31b69766992f41c5a3228cd3bf0e8d0e
I20260928 02:48:43.604732 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11040
I20260928 02:48:43.604682 11238 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:43 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:35478
I20260928 02:48:43.604718 11104 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:43 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:38442
I20260928 02:48:43.605190 11478 tablet_replica.cc:330] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: stopping tablet replica
I20260928 02:48:43.605484 11478 raft_consensus.cc:2229] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:48:43.605705 11372 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:43 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51446
I20260928 02:48:43.605800 11478 raft_consensus.cc:2258] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:43.606151 11479 tablet_replica.cc:330] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: stopping tablet replica
I20260928 02:48:43.606452 11479 raft_consensus.cc:2229] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:48:43.606742 11479 raft_consensus.cc:2258] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:43.607283 11478 ts_tablet_manager.cc:1905] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:43.608134 11479 ts_tablet_manager.cc:1905] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:43.609663 11478 ts_tablet_manager.cc:1918] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:43.609766 11478 log.cc:1196] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/31b69766992f41c5a3228cd3bf0e8d0e
I20260928 02:48:43.610064 11478 ts_tablet_manager.cc:1939] T 31b69766992f41c5a3228cd3bf0e8d0e P c3a6b366df78430c8f7b9dd2b0842d8a: Deleting consensus metadata
I20260928 02:48:43.611122 10962 catalog_manager.cc:4841] TS c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): tablet 31b69766992f41c5a3228cd3bf0e8d0e (table test-table [id=b44862b4a1304f069b0b0a84c4ace08c]) successfully deleted
I20260928 02:48:43.611423 11479 ts_tablet_manager.cc:1918] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:43.611537 11479 log.cc:1196] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/31b69766992f41c5a3228cd3bf0e8d0e
I20260928 02:48:43.611863 11479 ts_tablet_manager.cc:1939] T 31b69766992f41c5a3228cd3bf0e8d0e P 4f735704c39f4cb5ab2d8313890ea9d1: Deleting consensus metadata
I20260928 02:48:43.613011 10963 catalog_manager.cc:4841] TS 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): tablet 31b69766992f41c5a3228cd3bf0e8d0e (table test-table [id=b44862b4a1304f069b0b0a84c4ace08c]) successfully deleted
I20260928 02:48:43.613698 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11175
W20260928 02:48:43.613699 10963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:39493 (error 108)
I20260928 02:48:43.615192 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.614838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39493 (local address 127.0.0.1:38456)
0928 02:48:43.615004 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.615090 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:48:43.615358 10963 catalog_manager.cc:4584] TS 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31b69766992f41c5a3228cd3bf0e8d0e: Network error: Client connection negotiation failed: client connection to 127.28.248.129:39493: connect: Connection refused (error 111)
I20260928 02:48:43.620229 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11309
I20260928 02:48:43.625526 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 10946
I20260928 02:48:43.631695 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45511
--webserver_interface=127.28.248.190
--webserver_port=36855
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:43.738233 11481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:43.738541 11481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:43.738631 11481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:43.742645 11481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:43.742717 11481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:43.742758 11481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:43.742810 11481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:43.742851 11481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:43.747960 11481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:45511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=36855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:43.749001 11481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:43.749982 11481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:43.762449 11490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.762480 11487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.762578 11488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:43.762539 11481 server_base.cc:1034] running on GCE node
I20260928 02:48:43.763382 11481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:43.764526 11481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:43.765712 11481 hybrid_clock.cc:648] HybridClock initialized: now 1790563723765695 us; error 33 us; skew 500 ppm
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:43.768427 11481 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:43.769332 11481 webserver.cc:460] Webserver started at http://127.28.248.190:36855/ using document root <none> and password file <none>
I20260928 02:48:43.769873 11481 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:43.769939 11481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:43.773710 11481 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 02:48:43.775708 11497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.776927 11481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:48:43.777025 11481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "2782de666b8b44798b21906ab2e45f23"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "0a6cf6d32c8b620d775c9a1174eb4586"
server_key_iv: "4673c2eb51fa5b486854667654e842d1"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.777585 11481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:43.792064 11481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:43.794811 11481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:43.794965 11481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:43.801563 11481 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:45511
I20260928 02:48:43.801862 11549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:45511 every 8 connection(s)
I20260928 02:48:43.802646 11481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:43.806854 11550 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:48:43.807536 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11481
I20260928 02:48:43.808022 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:39493
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:45511
--webserver_port=41177
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:43.808975 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap starting.
I20260928 02:48:43.825685 11550 log.cc:824] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:43.834100 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:48:43.834447 11550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap complete.
I20260928 02:48:43.838409 11550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:43.838797 11550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2782de666b8b44798b21906ab2e45f23, State: Initialized, Role: FOLLOWER
I20260928 02:48:43.839286 11550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:43.839422 11550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:43.839506 11550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:43.839613 11550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:48:43.841451 11550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:43.841768 11550 leader_election.cc:302] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2782de666b8b44798b21906ab2e45f23; no voters:
I20260928 02:48:43.842097 11550 leader_election.cc:288] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:48:43.842173 11554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:48:43.842371 11554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 LEADER]: Becoming Leader. State: Replica: 2782de666b8b44798b21906ab2e45f23, State: Running, Role: LEADER
I20260928 02:48:43.842654 11554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:43.843259 11550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:43.843578 11556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2782de666b8b44798b21906ab2e45f23. Latest consensus state: current_term: 2 leader_uuid: "2782de666b8b44798b21906ab2e45f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } } }
I20260928 02:48:43.843703 11556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:43.843932 11555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2782de666b8b44798b21906ab2e45f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } } }
I20260928 02:48:43.844029 11555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:43.844043 11558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:43.848589 11558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:43.849196 11558 catalog_manager.cc:1259] Loaded cluster ID: 473252ef229d4468aedab94eba325db6
I20260928 02:48:43.849252 11558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:43.850648 11558 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:48:43.855575 11558 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:48:43.860689 11558 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:43.861346 11558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Loaded TSK: 0
I20260928 02:48:43.862460 11558 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:48:43.919422 11552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:43.919690 11552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:43.919760 11552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:43.923779 11552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:43.923851 11552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:43.923935 11552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:48:43.929011 11552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:39493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=41177
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:43.930003 11552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:43.930985 11552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:43.940274 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.940280 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:43.940286 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:43.940644 11552 server_base.cc:1034] running on GCE node
I20260928 02:48:43.941016 11552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:43.941653 11552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:43.942823 11552 hybrid_clock.cc:648] HybridClock initialized: now 1790563723942801 us; error 39 us; skew 500 ppm
Sep 28 02:48:43 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:43.945834 11552 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:48:43.946671 11552 webserver.cc:460] Webserver started at http://127.28.248.129:41177/ using document root <none> and password file <none>
I20260928 02:48:43.947198 11552 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:43.947299 11552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:43.951081 11552 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:48:43.953235 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:43.954618 11552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:48:43.954738 11552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "718927e002354f308fcd86723c8e9e48"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "7731396fae45839f3162e18a11fc35a8"
server_key_iv: "2042ca5196566c5aa6a7f95c28c9d2bc"
server_key_version: "encryptionkey@0"
I20260928 02:48:43.955281 11552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:43.965420 11552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:43.968259 11552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:43.968444 11552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:43.969049 11552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:43.970162 11594 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:48:43.975265 11552 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:48:43.975342 11552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 02:48:43.975420 11552 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:48:43.978008 11552 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:48:43.978070 11552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 02:48:43.978685 11594 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Bootstrap starting.
I20260928 02:48:43.988811 11552 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:39493
I20260928 02:48:43.988828 11701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:39493 every 8 connection(s)
I20260928 02:48:43.989775 11552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:48:43.994381 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11552
I20260928 02:48:43.994858 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:41951
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:45511
--webserver_port=40511
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:44.001025 11594 log.cc:824] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:44.009017 11594 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:48:44.009387 11594 tablet_bootstrap.cc:492] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Bootstrap complete.
I20260928 02:48:44.010496 11594 ts_tablet_manager.cc:1397] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.010s
I20260928 02:48:44.011878 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.992036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:57455)
0928 02:48:43.993589 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:43.993645 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:43.994842 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:43.996740 (+ 1898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:43.996755 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:43.997368 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:43.998028 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:43.998048 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.000347 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.000351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.001440 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.001451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.002523 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.002527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.002702 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.003410 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:44.003429 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:44.005469 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:44.008346 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.008351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.008363 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.008735 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.009179 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.009185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.009190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.009379 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.010175 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:44.010181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:44.010448 (+ 267us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.010632 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.010946 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":141,"threads_started":1}
I20260928 02:48:44.013013 11594 raft_consensus.cc:348] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:44.013528 11594 raft_consensus.cc:729] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718927e002354f308fcd86723c8e9e48, State: Initialized, Role: FOLLOWER
I20260928 02:48:44.013600 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:43.991990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:57455 (local address 127.28.248.190:45511)
0928 02:48:43.995365 (+ 3375us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:43.995389 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:43.995447 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:48:43.995781 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:43.995799 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:43.996238 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:43.998322 (+ 2084us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:43.998711 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.000162 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.001580 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.002151 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.002325 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.005709 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:44.005717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:44.005738 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:44.005780 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:44.008213 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.008869 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.008878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.008883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.009047 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.009520 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.009525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.009529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.009996 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:44.010121 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.012444 (+ 2323us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.012771 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3198,"thread_start_us":106,"threads_started":1}
I20260928 02:48:44.014022 11594 consensus_queue.cc:260] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } } peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } }
I20260928 02:48:44.014984 11594 ts_tablet_manager.cc:1428] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260928 02:48:44.015895 11702 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45511
I20260928 02:48:44.016113 11702 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:44.016567 11702 heartbeater.cc:511] Master 127.28.248.190:45511 requested a full tablet report, sending...
I20260928 02:48:44.019025 11514 ts_manager.cc:194] Registered new tserver with Master: 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493)
I20260928 02:48:44.023451 11514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:57455
I20260928 02:48:44.025885 11702 heartbeater.cc:503] Master 127.28.248.190:45511 was elected leader, sending a full tablet report...
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:48:44.032711 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.022338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39493 (local address 127.0.0.1:38468)
0928 02:48:44.022716 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.022757 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.022879 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.023482 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.023488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.023864 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:44.023968 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.023975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.025297 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.025301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.026181 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.026192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.026831 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.026835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.026952 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.027557 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:44.027572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:44.029080 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:44.030969 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.030978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.030983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.031212 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.031514 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.031517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.031519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.031602 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.032029 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:44.032036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:44.032168 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.032312 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.032501 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":122,"threads_started":1}
I20260928 02:48:44.032721 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.022393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.28.248.129:39493)
0928 02:48:44.023099 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.023107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.023124 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:48:44.023178 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.023182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.023327 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.023470 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:44.024085 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.025148 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.026289 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.026711 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.026830 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.029227 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:44.029236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:44.029251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:44.029271 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:44.030849 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.031326 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.031336 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.031337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.031411 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.031694 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.031697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.031699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.031915 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:44.032189 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.032392 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.032556 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":116,"threads_started":1}
I20260928 02:48:44.034057 11628 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b69766992f41c5a3228cd3bf0e8d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:43 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:38468
I20260928 02:48:44.034893 11711 tablet_replica.cc:330] stopping tablet replica
I20260928 02:48:44.035219 11711 raft_consensus.cc:2229] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:48:44.035495 11711 raft_consensus.cc:2258] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:44.036633 11711 ts_tablet_manager.cc:1905] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:44.041072 11711 ts_tablet_manager.cc:1918] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:44.041186 11711 log.cc:1196] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/31b69766992f41c5a3228cd3bf0e8d0e
I20260928 02:48:44.041479 11711 ts_tablet_manager.cc:1939] T 31b69766992f41c5a3228cd3bf0e8d0e P 718927e002354f308fcd86723c8e9e48: Deleting consensus metadata
I20260928 02:48:44.043349 11499 catalog_manager.cc:4841] TS 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493): tablet 31b69766992f41c5a3228cd3bf0e8d0e (table test-table [id=b44862b4a1304f069b0b0a84c4ace08c]) successfully deleted
W20260928 02:48:44.117923 11706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:44.118176 11706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:44.118218 11706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:44.122052 11706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:44.122097 11706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:44.122151 11706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:48:44.127185 11706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:41951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=40511
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:44.128093 11706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:44.129017 11706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:44.138132 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.138119 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.138119 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:44.138218 11706 server_base.cc:1034] running on GCE node
I20260928 02:48:44.138648 11706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:44.139248 11706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:44.140429 11706 hybrid_clock.cc:648] HybridClock initialized: now 1790563724140415 us; error 29 us; skew 500 ppm
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:44.143121 11706 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:48:44.143958 11706 webserver.cc:460] Webserver started at http://127.28.248.130:40511/ using document root <none> and password file <none>
I20260928 02:48:44.144507 11706 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:44.144605 11706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:44.148228 11706 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:48:44.150251 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.151449 11706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:48:44.151602 11706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "c3a6b366df78430c8f7b9dd2b0842d8a"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "9f11c6b0304ef59f86564e33fcbbe506"
server_key_iv: "8cc60d76f49ee216f681ab478ab4b2a1"
server_key_version: "encryptionkey@0"
I20260928 02:48:44.152122 11706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:44.171243 11706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:44.174492 11706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:44.174688 11706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:44.175259 11706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:44.176087 11706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:44.176152 11706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.176219 11706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:44.176266 11706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.185596 11706 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:41951
I20260928 02:48:44.185639 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:41951 every 8 connection(s)
I20260928 02:48:44.186913 11706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:48:44.195621 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11706
I20260928 02:48:44.196043 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:36985
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:45511
--webserver_port=42531
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:44.201743 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.188851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:34327 (local address 127.28.248.190:45511)
0928 02:48:44.189006 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.189010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.189866 (+ 856us) server_negotiation.cc:407] Connection header received
0928 02:48:44.191322 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.191335 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.191465 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.191568 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:44.192835 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.193416 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.194623 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.194901 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.195040 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.197650 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:44.197655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:44.197660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:44.197694 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:44.199467 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.199958 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.199965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.199970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.200069 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.200405 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.200412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.200415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.200727 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:44.200850 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.201308 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.201483 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:48:44.202270 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.189244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:34327)
0928 02:48:44.189715 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.189757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.191080 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.191754 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.191761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.192138 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:44.192687 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.192702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.193567 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.193571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.194493 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.194502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.195067 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.195072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.195251 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.195972 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:44.195989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:44.197485 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:44.199575 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.199579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.199590 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.199845 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.200169 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.200171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.200173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.200291 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.200835 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:44.200840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:44.201057 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.201199 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.201424 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":144,"threads_started":1}
I20260928 02:48:44.203429 11840 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45511
I20260928 02:48:44.203670 11840 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:44.204124 11840 heartbeater.cc:511] Master 127.28.248.190:45511 requested a full tablet report, sending...
I20260928 02:48:44.205194 11514 ts_manager.cc:194] Registered new tserver with Master: c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
I20260928 02:48:44.206151 11514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:34327
W20260928 02:48:44.317546 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:44.317818 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:44.317873 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:44.322220 11843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:44.322269 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:44.322325 11843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:48:44.328104 11843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:36985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=42531
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:44.329170 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:44.330121 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:44.339654 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:44.339720 11843 server_base.cc:1034] running on GCE node
W20260928 02:48:44.339668 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.339663 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:44.340287 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:44.340991 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:44.342146 11843 hybrid_clock.cc:648] HybridClock initialized: now 1790563724342113 us; error 43 us; skew 500 ppm
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:44.345184 11843 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:48:44.346071 11843 webserver.cc:460] Webserver started at http://127.28.248.131:42531/ using document root <none> and password file <none>
I20260928 02:48:44.346621 11843 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:44.346719 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:44.350418 11843 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:48:44.352365 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.353607 11843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:44.353740 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "8450c10a082d680738372108517c5a5a"
server_key_iv: "744225f6f01da6a41669e7e536852981"
server_key_version: "encryptionkey@0"
I20260928 02:48:44.354260 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:44.368916 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:44.371523 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:44.371699 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:44.372229 11843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:44.373061 11843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:44.373126 11843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.373184 11843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:44.373229 11843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.382071 11843 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:36985
I20260928 02:48:44.382151 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:36985 every 8 connection(s)
I20260928 02:48:44.383098 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:48:44.383682 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 11843
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:48:44.393776 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.384500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.28.248.190:45511)
0928 02:48:44.384662 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.384668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.384900 (+ 232us) server_negotiation.cc:407] Connection header received
0928 02:48:44.385054 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.385058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.385114 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.385198 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:44.386104 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.386929 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.387820 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.388055 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.388164 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.389529 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:44.389532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:44.389535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:44.389559 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:44.390950 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.391396 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.391402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.391406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.391514 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.392459 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.392474 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.392476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.392760 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:44.392976 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.393417 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.393557 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:48:44.397567 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.384982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54309 (local address 127.28.248.190:45511)
0928 02:48:44.385214 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.385218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.386137 (+ 919us) server_negotiation.cc:407] Connection header received
0928 02:48:44.387688 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.387691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.387737 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.387842 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:44.389100 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.389592 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.391089 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.391472 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.391571 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.394534 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:44.394537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:44.394538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:44.394581 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:44.395920 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.396346 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.396348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.396350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.396414 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:44.396729 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:44.396731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:44.396732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:44.396881 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:44.396953 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.397367 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.397454 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260928 02:48:44.398307 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.385497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:54309)
0928 02:48:44.385996 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.386055 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.387492 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.388013 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.388020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.388352 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:44.388967 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.388982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.389710 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.389714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.390914 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.390928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.391619 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.391623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.391795 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.392344 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:44.392361 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:44.394201 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:44.396014 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.396018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.396029 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.396265 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.396486 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:44.396488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:44.396489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:44.396648 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:44.396960 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:44.396965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:44.397172 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.397311 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.397537 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260928 02:48:44.399374 11972 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45511
I20260928 02:48:44.399587 11972 heartbeater.cc:464] Registering TS with master...
I20260928 02:48:44.399994 11972 heartbeater.cc:511] Master 127.28.248.190:45511 requested a full tablet report, sending...
I20260928 02:48:44.400887 11514 ts_manager.cc:194] Registered new tserver with Master: 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985)
I20260928 02:48:44.401407 11514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:54309
I20260928 02:48:44.406932 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:48:44.410663 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.407521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.28.248.190:45511)
0928 02:48:44.407632 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.407635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.407795 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:48:44.407915 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.407918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.407997 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.408083 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:44.408259 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.408779 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.409552 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.409762 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.409834 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.410072 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:44.410390 (+ 318us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:44.410471 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.410532 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.410576 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:48:44.411199 11514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:48:44.414337 11514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:48:44.421633 11628 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf7a1c2b7f24200a47f20ba46bbb884 (DEFAULT_TABLE table=test-table [id=f101dc810bd34997b6563313d64f5f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:44.422271 11628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf7a1c2b7f24200a47f20ba46bbb884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:44.425197 11594 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Bootstrap starting.
I20260928 02:48:44.426064 11594 tablet_bootstrap.cc:654] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:44.426291 11709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.420531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:36985 (local address 127.0.0.1:34632)
0928 02:48:44.420669 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.420684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.420755 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.421682 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.421685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.421703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:44.421822 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.421830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.423536 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.423541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.424778 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.424788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.425838 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.425841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.425939 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.425994 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.426072 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.426158 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:48:44.426389 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.420709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.28.248.131:36985)
0928 02:48:44.421046 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.421059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.421084 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:48:44.421164 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.421170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.421504 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.421985 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:44.422142 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.423414 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.424928 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.425721 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.425844 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.425993 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.426126 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.426234 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":134,"threads_started":1}
I20260928 02:48:44.427047 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.421013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:41951 (local address 127.0.0.1:42954)
0928 02:48:44.421302 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.421339 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.421425 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.422276 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.422279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.422297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:44.422384 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.422390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.423759 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.423762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.424897 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.424904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.426659 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.426663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.426759 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.426781 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.426851 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.426921 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260928 02:48:44.427239 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.421430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.28.248.130:41951)
0928 02:48:44.421802 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.421815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.421839 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:48:44.421911 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.421916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.422107 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.422290 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:44.422495 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.423603 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.425011 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.426267 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.426670 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.426792 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.426875 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.426957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":158,"threads_started":1}
I20260928 02:48:44.428059 11594 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: No bootstrap required, opened a new log
I20260928 02:48:44.428160 11594 ts_tablet_manager.cc:1397] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:48:44.428599 11594 raft_consensus.cc:348] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.428748 11594 raft_consensus.cc:374] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:44.428793 11594 raft_consensus.cc:729] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718927e002354f308fcd86723c8e9e48, State: Initialized, Role: FOLLOWER
I20260928 02:48:44.428912 11594 consensus_queue.cc:260] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.429778 11594 ts_tablet_manager.cc:1428] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:48:44.429842 11774 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf7a1c2b7f24200a47f20ba46bbb884 (DEFAULT_TABLE table=test-table [id=f101dc810bd34997b6563313d64f5f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:44.429843 11906 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf7a1c2b7f24200a47f20ba46bbb884 (DEFAULT_TABLE table=test-table [id=f101dc810bd34997b6563313d64f5f9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:48:44.430904 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf7a1c2b7f24200a47f20ba46bbb884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:44.430930 11906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf7a1c2b7f24200a47f20ba46bbb884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:44.437433 11986 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Bootstrap starting.
I20260928 02:48:44.437427 11985 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Bootstrap starting.
I20260928 02:48:44.439900 11985 tablet_bootstrap.cc:654] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:44.439900 11986 tablet_bootstrap.cc:654] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:44.440587 11985 log.cc:824] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:44.440588 11986 log.cc:824] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:44.442276 11986 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: No bootstrap required, opened a new log
I20260928 02:48:44.442286 11985 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: No bootstrap required, opened a new log
I20260928 02:48:44.442480 11985 ts_tablet_manager.cc:1397] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:48:44.442487 11986 ts_tablet_manager.cc:1397] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 02:48:44.445674 11985 raft_consensus.cc:348] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.445674 11986 raft_consensus.cc:348] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.445897 11985 raft_consensus.cc:374] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:44.445891 11986 raft_consensus.cc:374] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:44.445955 11986 raft_consensus.cc:729] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f735704c39f4cb5ab2d8313890ea9d1, State: Initialized, Role: FOLLOWER
I20260928 02:48:44.445955 11985 raft_consensus.cc:729] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a6b366df78430c8f7b9dd2b0842d8a, State: Initialized, Role: FOLLOWER
I20260928 02:48:44.446367 11985 consensus_queue.cc:260] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.446373 11986 consensus_queue.cc:260] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.446964 11840 heartbeater.cc:503] Master 127.28.248.190:45511 was elected leader, sending a full tablet report...
I20260928 02:48:44.446965 11972 heartbeater.cc:503] Master 127.28.248.190:45511 was elected leader, sending a full tablet report...
I20260928 02:48:44.447309 11985 ts_tablet_manager.cc:1428] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:48:44.447309 11986 ts_tablet_manager.cc:1428] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 02:48:44.474799 11989 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:44.474993 11989 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.476114 11989 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493)
I20260928 02:48:44.480103 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.476562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:49969 (local address 127.28.248.131:36985)
0928 02:48:44.476730 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.476735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.476749 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:48:44.476903 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.476906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.476956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.477039 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:44.477380 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.477974 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.479076 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.479524 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.479614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.479653 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.479878 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.479952 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:48:44.480109 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.476385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:36985 (local address 127.28.248.130:49969)
0928 02:48:44.476658 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.476675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.476769 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.477084 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.477087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.477111 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:44.477250 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.477258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.478080 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.478083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.478956 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.478964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.479630 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.479634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.479746 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.479778 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.479850 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.479937 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":30}
I20260928 02:48:44.480361 11710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.476782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:41995 (local address 127.28.248.129:39493)
0928 02:48:44.476911 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.476922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.477084 (+ 162us) server_negotiation.cc:407] Connection header received
0928 02:48:44.477194 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.477197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.477258 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.477356 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:48:44.477616 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.478165 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.479215 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.479695 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.479811 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.479964 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.480091 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.480179 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:48:44.480710 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.476660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39493 (local address 127.28.248.130:41995)
0928 02:48:44.476989 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.477005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.477096 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:44.477374 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:44.477376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:44.477392 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:48:44.477505 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.477510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.478270 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.478272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.479096 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:44.479105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.479805 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.479808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:44.479921 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.479942 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:48:44.480490 (+ 548us) client_negotiation.cc:240] Negotiation successful
0928 02:48:44.480565 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":16,"thread_start_us":118,"threads_started":1}
I20260928 02:48:44.480799 11926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf7a1c2b7f24200a47f20ba46bbb884" candidate_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" is_pre_election: true
I20260928 02:48:44.481088 11926 raft_consensus.cc:2454] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a6b366df78430c8f7b9dd2b0842d8a in term 0.
I20260928 02:48:44.481302 11656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf7a1c2b7f24200a47f20ba46bbb884" candidate_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718927e002354f308fcd86723c8e9e48" is_pre_election: true
I20260928 02:48:44.481568 11656 raft_consensus.cc:2454] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3a6b366df78430c8f7b9dd2b0842d8a in term 0.
I20260928 02:48:44.481588 11728 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a; no voters:
I20260928 02:48:44.481834 11989 raft_consensus.cc:2781] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:48:44.481918 11989 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:48:44.481977 11989 raft_consensus.cc:3037] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:44.483127 11989 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.483505 11989 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [CANDIDATE]: Term 1 election: Requested vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493)
I20260928 02:48:44.483839 11926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf7a1c2b7f24200a47f20ba46bbb884" candidate_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
I20260928 02:48:44.483865 11656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf7a1c2b7f24200a47f20ba46bbb884" candidate_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718927e002354f308fcd86723c8e9e48"
I20260928 02:48:44.483929 11926 raft_consensus.cc:3037] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:44.483996 11656 raft_consensus.cc:3037] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:44.485005 11926 raft_consensus.cc:2454] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a6b366df78430c8f7b9dd2b0842d8a in term 1.
I20260928 02:48:44.485076 11656 raft_consensus.cc:2454] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3a6b366df78430c8f7b9dd2b0842d8a in term 1.
I20260928 02:48:44.485293 11728 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a; no voters:
I20260928 02:48:44.485455 11989 raft_consensus.cc:2781] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:44.485733 11989 raft_consensus.cc:686] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 LEADER]: Becoming Leader. State: Replica: c3a6b366df78430c8f7b9dd2b0842d8a, State: Running, Role: LEADER
I20260928 02:48:44.486106 11989 consensus_queue.cc:237] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:44.489480 11514 catalog_manager.cc:5495] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130). New cstate: current_term: 1 leader_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } health_report { overall_health: UNKNOWN } } }
W20260928 02:48:44.493690 11703 tablet.cc:2369] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:48:44.505774 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.502626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.28.248.130:41951)
0928 02:48:44.502744 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:44.502748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:44.502760 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:48:44.502834 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:44.502836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:44.502887 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:44.502974 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:44.503153 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.503619 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.504364 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:44.504565 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:44.504714 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:44.504978 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:44.505355 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:44.505443 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:44.505563 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:48:44.505611 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:48:44.511282 11656 raft_consensus.cc:1261] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Refusing update from remote peer c3a6b366df78430c8f7b9dd2b0842d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:44.511312 11926 raft_consensus.cc:1261] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Refusing update from remote peer c3a6b366df78430c8f7b9dd2b0842d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:44.511936 11989 consensus_queue.cc:1035] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:44.512128 11992 consensus_queue.cc:1035] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:44.518044 11995 mvcc.cc:205] Tried to move back new op lower bound from 7334149015591505920 to 7334149015498330112. Current Snapshot: MvccSnapshot[applied={T|T < 7334149015591505920}]
I20260928 02:48:44.518740 11996 mvcc.cc:205] Tried to move back new op lower bound from 7334149015591505920 to 7334149015498330112. Current Snapshot: MvccSnapshot[applied={T|T < 7334149015591505920}]
I20260928 02:48:44.520334 11997 mvcc.cc:205] Tried to move back new op lower bound from 7334149015591505920 to 7334149015498330112. Current Snapshot: MvccSnapshot[applied={T|T < 7334149015591505920}]
I20260928 02:48:44.526012 11514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:48:44.526178 11514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:48:44.528393 11514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Sending DeleteTablet for 3 replicas of tablet 5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:44.529096 11906 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf7a1c2b7f24200a47f20ba46bbb884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:44 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:34632
I20260928 02:48:44.529119 11774 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf7a1c2b7f24200a47f20ba46bbb884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:44 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:42954
I20260928 02:48:44.529560 12005 tablet_replica.cc:330] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: stopping tablet replica
I20260928 02:48:44.529745 11628 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf7a1c2b7f24200a47f20ba46bbb884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:48:44 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:38468
I20260928 02:48:44.529852 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11552
I20260928 02:48:44.529917 11711 tablet_replica.cc:330] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: stopping tablet replica
I20260928 02:48:44.530054 11711 raft_consensus.cc:2229] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:48:44.530179 11711 raft_consensus.cc:2258] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:44.530759 12005 raft_consensus.cc:2229] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:48:44.531082 12006 tablet_replica.cc:330] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: stopping tablet replica
I20260928 02:48:44.531118 12005 raft_consensus.cc:2258] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:44.531358 12006 raft_consensus.cc:2229] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:48:44.531741 12006 raft_consensus.cc:2258] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:48:44.532977 12005 ts_tablet_manager.cc:1905] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:44.533900 12006 ts_tablet_manager.cc:1905] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:48:44.536466 12005 ts_tablet_manager.cc:1918] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:44.536672 12005 log.cc:1196] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:44.536747 12006 ts_tablet_manager.cc:1918] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:48:44.536842 12006 log.cc:1196] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:44.537195 12006 ts_tablet_manager.cc:1939] T 5cf7a1c2b7f24200a47f20ba46bbb884 P c3a6b366df78430c8f7b9dd2b0842d8a: Deleting consensus metadata
I20260928 02:48:44.537199 12005 ts_tablet_manager.cc:1939] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 4f735704c39f4cb5ab2d8313890ea9d1: Deleting consensus metadata
W20260928 02:48:44.537876 11499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:39493 (error 108)
I20260928 02:48:44.538370 11498 catalog_manager.cc:4841] TS c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): tablet 5cf7a1c2b7f24200a47f20ba46bbb884 (table test-table [id=f101dc810bd34997b6563313d64f5f9e]) successfully deleted
I20260928 02:48:44.538431 11499 catalog_manager.cc:4841] TS 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): tablet 5cf7a1c2b7f24200a47f20ba46bbb884 (table test-table [id=f101dc810bd34997b6563313d64f5f9e]) successfully deleted
I20260928 02:48:44.539031 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11706
I20260928 02:48:44.540910 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:44.540646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:39493 (local address 127.0.0.1:52520)
0928 02:48:44.540775 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.540840 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:48:44.541126 11499 catalog_manager.cc:4584] TS 718927e002354f308fcd86723c8e9e48 (127.28.248.129:39493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cf7a1c2b7f24200a47f20ba46bbb884: Network error: Client connection negotiation failed: client connection to 127.28.248.129:39493: connect: Connection refused (error 111)
I20260928 02:48:44.544498 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11843
I20260928 02:48:44.550307 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 11481
I20260928 02:48:44.556550 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45511
--webserver_interface=127.28.248.190
--webserver_port=36855
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:44.664273 12008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:44.664572 12008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:44.664674 12008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:44.668589 12008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:44.668691 12008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:44.668730 12008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:44.668764 12008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:44.668795 12008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:44.674007 12008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:45511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=36855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:44.675086 12008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:44.676090 12008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:44.684229 12016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.684202 12014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.684351 12013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:44.684702 12008 server_base.cc:1034] running on GCE node
I20260928 02:48:44.685220 12008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:44.686291 12008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:44.687467 12008 hybrid_clock.cc:648] HybridClock initialized: now 1790563724687439 us; error 47 us; skew 500 ppm
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:44.690037 12008 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:48:44.690908 12008 webserver.cc:460] Webserver started at http://127.28.248.190:36855/ using document root <none> and password file <none>
I20260928 02:48:44.691442 12008 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:44.691546 12008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:44.695127 12008 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:44.697118 12023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.698275 12008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:44.698401 12008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "2782de666b8b44798b21906ab2e45f23"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "0a6cf6d32c8b620d775c9a1174eb4586"
server_key_iv: "4673c2eb51fa5b486854667654e842d1"
server_key_version: "encryptionkey@0"
I20260928 02:48:44.698927 12008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:44.712697 12008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:44.715277 12008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:44.715464 12008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:44.722172 12008 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:45511
I20260928 02:48:44.722445 12075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:45511 every 8 connection(s)
I20260928 02:48:44.723268 12008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:44.727687 12076 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:48:44.729300 12076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap starting.
I20260928 02:48:44.732117 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12008
I20260928 02:48:44.732533 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:39493
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:45511
--webserver_port=41177
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:44.751370 12076 log.cc:824] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:44.762748 12076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:48:44.763159 12076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23: Bootstrap complete.
I20260928 02:48:44.767246 12076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:44.767719 12076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2782de666b8b44798b21906ab2e45f23, State: Initialized, Role: FOLLOWER
I20260928 02:48:44.768210 12076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:44.768319 12076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:44.768401 12076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:44.768568 12076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:48:44.770553 12076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:44.770886 12076 leader_election.cc:302] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2782de666b8b44798b21906ab2e45f23; no voters:
I20260928 02:48:44.771229 12076 leader_election.cc:288] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:48:44.771363 12080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:48:44.771638 12080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [term 3 LEADER]: Becoming Leader. State: Replica: 2782de666b8b44798b21906ab2e45f23, State: Running, Role: LEADER
I20260928 02:48:44.771951 12080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } }
I20260928 02:48:44.772482 12076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:44.773494 12082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2782de666b8b44798b21906ab2e45f23. Latest consensus state: current_term: 3 leader_uuid: "2782de666b8b44798b21906ab2e45f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } } }
I20260928 02:48:44.773500 12081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2782de666b8b44798b21906ab2e45f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782de666b8b44798b21906ab2e45f23" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45511 } } }
I20260928 02:48:44.773618 12082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:44.773631 12081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782de666b8b44798b21906ab2e45f23 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:44.773952 12089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:44.779402 12089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:44.780083 12089 catalog_manager.cc:1259] Loaded cluster ID: 473252ef229d4468aedab94eba325db6
I20260928 02:48:44.780138 12089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:48:44.781168 12089 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0d0e1f09ad google::LogMessage::Fail() at ??:0
@ 0x7f0d0e1f388a google::LogMessage::SendToLog() at ??:0
@ 0x7f0d0e1f04de google::LogMessage::Flush() at ??:0
@ 0x7f0d0e1f1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0d18b2ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0d18b30d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0d18b2d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0d18b71640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0d104423d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0d10442c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0d104443f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0d10433512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0d11a776db start_thread at ??:0
@ 0x7f0d0ccee61f clone at ??:0
W20260928 02:48:44.883363 12078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:44.883740 12078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:44.883823 12078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:44.889844 12078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:44.889923 12078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:44.890009 12078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:48:44.898097 12078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:39493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=41177
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:44.899426 12078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:44.900888 12078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:44.911947 12117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.911931 12120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:44.911916 12118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:44.912395 12078 server_base.cc:1034] running on GCE node
I20260928 02:48:44.912837 12078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:44.913565 12078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:44.914769 12078 hybrid_clock.cc:648] HybridClock initialized: now 1790563724914730 us; error 46 us; skew 500 ppm
Sep 28 02:48:44 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:44.918146 12078 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:48:44.919234 12078 webserver.cc:460] Webserver started at http://127.28.248.129:41177/ using document root <none> and password file <none>
I20260928 02:48:44.919912 12078 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:44.920001 12078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:44.925271 12078 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:48:44.927897 12127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:44.929468 12078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:48:44.929584 12078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "718927e002354f308fcd86723c8e9e48"
format_stamp: "Formatted at 2026-09-28 02:48:42 on dist-test-slave-r238"
server_key: "7731396fae45839f3162e18a11fc35a8"
server_key_iv: "2042ca5196566c5aa6a7f95c28c9d2bc"
server_key_version: "encryptionkey@0"
I20260928 02:48:44.930274 12078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:44.961938 12078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:44.965821 12078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:44.966082 12078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:44.966879 12078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:44.968357 12134 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:48:44.972499 12078 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:48:44.972584 12078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260928 02:48:44.972678 12078 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:48:44.975029 12078 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:48:44.975102 12078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:48:44.975503 12134 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Bootstrap starting.
I20260928 02:48:44.990023 12078 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:39493
I20260928 02:48:44.990039 12241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:39493 every 8 connection(s)
I20260928 02:48:44.991680 12078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:48:44.995703 12134 log.cc:824] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:45.000988 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12078
I20260928 02:48:45.001408 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:41951
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:45511
--webserver_port=40511
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:45.004503 12134 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:48:45.004942 12134 tablet_bootstrap.cc:492] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Bootstrap complete.
I20260928 02:48:45.006435 12134 ts_tablet_manager.cc:1397] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.000s
I20260928 02:48:45.010015 12134 raft_consensus.cc:348] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:45.010588 12134 raft_consensus.cc:729] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718927e002354f308fcd86723c8e9e48, State: Initialized, Role: FOLLOWER
I20260928 02:48:45.011206 12134 consensus_queue.cc:260] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:45.012723 12134 ts_tablet_manager.cc:1428] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260928 02:48:45.044768 12244 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:44.994126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:41317)
0928 02:48:44.994739 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:44.994797 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:44.995872 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:45.043952 (+ 48080us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":147,"threads_started":1}
W20260928 02:48:45.045248 12242 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:45511 (0 consecutive failures): Network error: Failed to ping master at 127.28.248.190:45511: Client connection negotiation failed: client connection to 127.28.248.190:45511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:48:45.045979 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.045748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:46783)
0928 02:48:45.045867 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.045925 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:48:45.046612 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.046416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:45545)
0928 02:48:45.046512 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.046563 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:48:45.046772 12242 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:48:45.121647 12246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:45.121985 12246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:45.122087 12246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:45.126109 12246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:45.126185 12246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:45.126268 12246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:48:45.131724 12246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:41951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=40511
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:45.132925 12246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:45.133975 12246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:45.143885 12252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:45.143998 12246 server_base.cc:1034] running on GCE node
W20260928 02:48:45.143947 12253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:45.143893 12255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:45.144667 12246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:45.145357 12246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:45.146533 12246 hybrid_clock.cc:648] HybridClock initialized: now 1790563725146498 us; error 45 us; skew 500 ppm
Sep 28 02:48:45 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:45.149451 12246 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:48:45.150295 12246 webserver.cc:460] Webserver started at http://127.28.248.130:40511/ using document root <none> and password file <none>
I20260928 02:48:45.150861 12246 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:45.150949 12246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:45.154780 12246 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:48:45.156804 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:45.158052 12246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:48:45.158170 12246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "c3a6b366df78430c8f7b9dd2b0842d8a"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "9f11c6b0304ef59f86564e33fcbbe506"
server_key_iv: "8cc60d76f49ee216f681ab478ab4b2a1"
server_key_version: "encryptionkey@0"
I20260928 02:48:45.158701 12246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:45.180536 12246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:45.183514 12246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:45.183720 12246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:45.184329 12246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:45.185292 12246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:45.185359 12246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:45.185426 12246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:45.185472 12246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:45.194275 12246 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:41951
I20260928 02:48:45.194453 12375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:41951 every 8 connection(s)
I20260928 02:48:45.195487 12246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:48:45.198696 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12246
I20260928 02:48:45.199147 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:36985
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:45511
--webserver_port=42531
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:45.199883 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.197764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:44163)
0928 02:48:45.198234 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.198819 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
W20260928 02:48:45.200364 12376 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:45511 (0 consecutive failures): Network error: Failed to ping master at 127.28.248.190:45511: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
I20260928 02:48:45.201161 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.200894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:40673)
0928 02:48:45.201037 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.201102 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:48:45.201900 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.201672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:51413)
0928 02:48:45.201784 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.201845 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:48:45.202080 12376 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:48:45.310889 12379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:45.311203 12379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:45.311283 12379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:45.315653 12379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:45.315742 12379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:45.315840 12379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:48:45.321357 12379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:36985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=42531
--tserver_master_addrs=127.28.248.190:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:48:45.322465 12379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:45.323457 12379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:45.332990 12384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:45.333106 12379 server_base.cc:1034] running on GCE node
W20260928 02:48:45.332985 12385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:45.333031 12387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:45.333643 12379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:45.334286 12379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:45.335455 12379 hybrid_clock.cc:648] HybridClock initialized: now 1790563725335433 us; error 40 us; skew 500 ppm
Sep 28 02:48:45 dist-test-slave-r238 krb5kdc[10933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:45.338145 12379 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:48:45.338959 12379 webserver.cc:460] Webserver started at http://127.28.248.131:42531/ using document root <none> and password file <none>
I20260928 02:48:45.339515 12379 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:45.339601 12379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:45.343330 12379 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:45.345433 12394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:45.346617 12379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:48:45.346741 12379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "4f735704c39f4cb5ab2d8313890ea9d1"
format_stamp: "Formatted at 2026-09-28 02:48:43 on dist-test-slave-r238"
server_key: "8450c10a082d680738372108517c5a5a"
server_key_iv: "744225f6f01da6a41669e7e536852981"
server_key_version: "encryptionkey@0"
I20260928 02:48:45.347280 12379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:45.365578 12379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:45.368593 12379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:45.368836 12379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:45.369441 12379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:48:45.370277 12379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:48:45.370345 12379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:45.370414 12379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:48:45.370460 12379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:45.379666 12379 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:36985
I20260928 02:48:45.379688 12507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:36985 every 8 connection(s)
I20260928 02:48:45.380748 12379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:48:45.384399 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.382811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:44449)
0928 02:48:45.383275 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.383679 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
W20260928 02:48:45.384855 12508 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:45511 (0 consecutive failures): Network error: Failed to ping master at 127.28.248.190:45511: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
I20260928 02:48:45.385478 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.385277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:60603)
0928 02:48:45.385380 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.385425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:48:45.386016 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12379
I20260928 02:48:45.386052 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:45.385870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:38635)
0928 02:48:45.385960 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:45.385998 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:48:45.386219 12508 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:48:46.047854 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.047373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:40471)
0928 02:48:46.047673 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:46.047733 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260928 02:48:46.203111 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.202646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:34411)
0928 02:48:46.202915 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:46.202980 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260928 02:48:46.387161 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.386759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:50493)
0928 02:48:46.386996 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:46.387067 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260928 02:48:46.753328 12515 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:46.753510 12515 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:46.754582 12515 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
Sep 28 02:48:46 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.130@KRBTEST.COM
Sep 28 02:48:46 dist-test-slave-r238 krb5kdc[10933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.131@KRBTEST.COM
I20260928 02:48:46.768688 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.754792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:36985 (local address 127.28.248.129:49257)
0928 02:48:46.755071 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:46.755089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:46.755194 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:46.756964 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:46.756968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:46.757587 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:46.758299 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:46.758320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.759617 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.759621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:46.760581 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:46.760591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.761327 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.761331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:46.761543 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:46.762257 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:46.762269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:46.764145 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:46.766356 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:46.766359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:46.766361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:46.766615 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:46.766929 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:46.766931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:46.766933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:46.767051 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:46.767787 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:46.767793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:46.768082 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 02:48:46.768270 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 02:48:46.768493 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":198,"spinlock_wait_cycles":12672,"thread_start_us":60,"threads_started":1}
I20260928 02:48:46.768812 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.754988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:49257 (local address 127.28.248.131:36985)
0928 02:48:46.755324 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:46.755353 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:46.755403 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:48:46.756230 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:46.756237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:46.756607 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:46.757567 (+ 960us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:46.758495 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.759476 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.760751 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.761200 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.761366 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:46.764343 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:46.764349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:46.764366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:46.764392 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:46.766258 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:46.766703 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:46.766708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:46.766712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:46.766841 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:46.767155 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:46.767158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:46.767160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:46.767659 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:46.767785 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:46.768379 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 02:48:46.768597 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260928 02:48:46.769203 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.754906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:41951 (local address 127.28.248.129:51277)
0928 02:48:46.755429 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:46.755442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:46.755534 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:46.756961 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:46.756966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:46.757584 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:46.758299 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:46.758321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.759643 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.759646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:46.760583 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:46.760600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.761327 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.761330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:46.761544 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:46.762200 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:46.762215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:46.763756 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:46.766133 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:46.766138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:46.766148 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:46.766494 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:46.766862 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:46.766864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:46.766865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:46.767068 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:46.768691 (+ 1623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:46.768696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:46.768787 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:48:46.768913 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:48:46.769049 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":171,"spinlock_wait_cycles":18304,"thread_start_us":91,"threads_started":1}
I20260928 02:48:46.769515 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:46.755110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:51277 (local address 127.28.248.130:41951)
0928 02:48:46.755458 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:46.755477 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:46.755529 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:48:46.756233 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:46.756239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:46.756581 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:46.756918 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:46.758507 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.759472 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.760752 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:46.761199 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:46.761367 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:46.763976 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:46.763981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:46.763993 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:46.764021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:46.766018 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:46.766607 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:46.766612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:46.766616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:46.766771 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:46.767155 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:46.767158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:46.767160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:46.767659 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:46.767785 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:46.769040 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0928 02:48:46.769300 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
W20260928 02:48:46.771350 12129 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
W20260928 02:48:46.771512 12128 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:46.771616 12128 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 718927e002354f308fcd86723c8e9e48; no voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a
I20260928 02:48:46.771876 12515 raft_consensus.cc:2726] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:48:47.048890 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:47.048592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:54685)
0928 02:48:47.048747 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:47.048804 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:48:47.204275 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:47.203831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:56685)
0928 02:48:47.204095 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:47.204171 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260928 02:48:47.388302 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:47.387867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:42161)
0928 02:48:47.388128 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:47.388202 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
I20260928 02:48:48.049975 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:48.049568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:51855)
0928 02:48:48.049828 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:48.049882 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260928 02:48:48.205382 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:48.205027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:59787)
0928 02:48:48.205246 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:48.205298 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260928 02:48:48.389572 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:48.389044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:52879)
0928 02:48:48.389373 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:48.389463 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260928 02:48:48.615840 12525 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:48.615970 12525 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:48.616379 12525 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
W20260928 02:48:48.617180 12129 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
W20260928 02:48:48.617285 12128 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:48.617380 12128 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 718927e002354f308fcd86723c8e9e48; no voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a
I20260928 02:48:48.617542 12525 raft_consensus.cc:2726] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:48:49.051136 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:49.050740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:60299)
0928 02:48:49.050973 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:49.051029 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260928 02:48:49.206703 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:49.206204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:40019)
0928 02:48:49.206544 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:49.206599 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":152,"threads_started":1}
I20260928 02:48:49.390729 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:49.390328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:56127)
0928 02:48:49.390582 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:49.390639 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260928 02:48:50.052548 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:50.052085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:51747)
0928 02:48:50.052370 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:50.052430 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260928 02:48:50.207937 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:50.207539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:44993)
0928 02:48:50.207795 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:50.207847 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260928 02:48:50.391804 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:50.391431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:46181)
0928 02:48:50.391669 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:50.391719 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260928 02:48:51.053742 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:51.053344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:59355)
0928 02:48:51.053595 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:51.053647 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260928 02:48:51.209157 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:51.208764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:55363)
0928 02:48:51.209009 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:51.209066 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260928 02:48:51.392988 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:51.392551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:45457)
0928 02:48:51.392850 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:51.392902 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":141,"threads_started":1}
I20260928 02:48:51.868342 12535 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:51.868470 12535 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:51.868911 12535 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
W20260928 02:48:51.869700 12128 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
W20260928 02:48:51.869798 12129 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:51.869932 12129 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 718927e002354f308fcd86723c8e9e48; no voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a
I20260928 02:48:51.870150 12535 raft_consensus.cc:2726] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:48:52.055120 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:52.054560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:43433)
0928 02:48:52.054917 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:52.054991 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260928 02:48:52.210266 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:52.209859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:46523)
0928 02:48:52.210129 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:52.210179 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":67,"threads_started":1}
I20260928 02:48:52.394150 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:52.393696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:36375)
0928 02:48:52.394003 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:52.394059 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260928 02:48:53.056367 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:53.055960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:47687)
0928 02:48:53.056216 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:53.056271 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260928 02:48:53.211493 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:53.211030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:58883)
0928 02:48:53.211304 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:53.211359 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260928 02:48:53.395291 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:53.394913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:59397)
0928 02:48:53.395145 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:53.395205 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260928 02:48:54.057574 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:54.057138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:35603)
0928 02:48:54.057413 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:54.057473 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260928 02:48:54.212759 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:54.212280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:41253)
0928 02:48:54.212552 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:54.212657 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260928 02:48:54.396803 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:54.396323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:46025)
0928 02:48:54.396597 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:54.396712 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260928 02:48:55.058913 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:55.058483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:54771)
0928 02:48:55.058756 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:55.058817 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260928 02:48:55.213891 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:55.213500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:58679)
0928 02:48:55.213746 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:55.213800 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260928 02:48:55.397881 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:55.397493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:48163)
0928 02:48:55.397738 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:55.397789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260928 02:48:56.060084 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:56.059668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:47991)
0928 02:48:56.059911 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:56.059966 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260928 02:48:56.215121 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:56.214686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:40365)
0928 02:48:56.214972 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:56.215027 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
I20260928 02:48:56.399022 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:56.398639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:50761)
0928 02:48:56.398887 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:56.398935 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260928 02:48:56.846550 12551 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:48:56.846683 12551 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:48:56.847117 12551 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
W20260928 02:48:56.847863 12129 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
W20260928 02:48:56.847965 12128 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:48:56.848076 12128 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 718927e002354f308fcd86723c8e9e48; no voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a
I20260928 02:48:56.848261 12551 raft_consensus.cc:2726] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:48:57.061467 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:57.061043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:41447)
0928 02:48:57.061310 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:57.061369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260928 02:48:57.216576 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:57.215966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:43081)
0928 02:48:57.216210 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:57.216266 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260928 02:48:57.400121 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:57.399719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:55537)
0928 02:48:57.399961 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:57.400014 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260928 02:48:58.062708 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:58.062305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:51683)
0928 02:48:58.062563 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:58.062615 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260928 02:48:58.217830 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:58.217450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:51139)
0928 02:48:58.217683 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:58.217738 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260928 02:48:58.401347 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:58.400947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:55065)
0928 02:48:58.401202 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:58.401256 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260928 02:48:59.063942 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:59.063539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:40447)
0928 02:48:59.063789 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:59.063842 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260928 02:48:59.218937 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:59.218534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:37207)
0928 02:48:59.218789 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:59.218842 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260928 02:48:59.402495 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:59.402112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:54067)
0928 02:48:59.402349 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:59.402404 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260928 02:49:00.065364 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:00.064876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.129:40367)
0928 02:49:00.065133 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:00.065221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260928 02:49:00.188295 12562 raft_consensus.cc:482] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:00.188422 12562 raft_consensus.cc:504] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f735704c39f4cb5ab2d8313890ea9d1" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 36985 } } peers { permanent_uuid: "c3a6b366df78430c8f7b9dd2b0842d8a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 41951 } } peers { permanent_uuid: "718927e002354f308fcd86723c8e9e48" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 39493 } }
I20260928 02:49:00.189005 12562 leader_election.cc:288] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985), c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951)
W20260928 02:49:00.189734 12128 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3a6b366df78430c8f7b9dd2b0842d8a (127.28.248.130:41951): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
W20260928 02:49:00.189833 12129 leader_election.cc:341] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f735704c39f4cb5ab2d8313890ea9d1 (127.28.248.131:36985): Not found: Tablet not found: 5cf7a1c2b7f24200a47f20ba46bbb884
I20260928 02:49:00.189893 12129 leader_election.cc:302] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 718927e002354f308fcd86723c8e9e48; no voters: 4f735704c39f4cb5ab2d8313890ea9d1, c3a6b366df78430c8f7b9dd2b0842d8a
I20260928 02:49:00.190027 12562 raft_consensus.cc:2726] T 5cf7a1c2b7f24200a47f20ba46bbb884 P 718927e002354f308fcd86723c8e9e48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:49:00.220209 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:00.219791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.130:43423)
0928 02:49:00.220041 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:00.220120 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260928 02:49:00.393833 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12078
I20260928 02:49:00.399660 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12246
I20260928 02:49:00.403813 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:00.403404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45511 (local address 127.28.248.131:58703)
0928 02:49:00.403676 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:00.403728 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:45511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260928 02:49:00.405189 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12379
2026-09-28T02:49:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19075 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:49:00 dist-test-slave-r238 krb5kdc[12567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:49:00 dist-test-slave-r238 krb5kdc[12567](info): set up 2 sockets
Sep 28 02:49:00 dist-test-slave-r238 krb5kdc[12567](info): commencing operation
krb5kdc: starting...
W20260928 02:49:01.453486 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:49:01 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:49:01.579882 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36499
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:36499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:49:01.690934 12580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:01.691243 12580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:01.691286 12580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:01.695289 12580 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:49:01.695351 12580 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:49:01.695366 12580 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:49:01.695381 12580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:49:01.695399 12580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:01.695417 12580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:01.695433 12580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:49:01.695448 12580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:49:01.700815 12580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:36499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:01.701890 12580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:01.702927 12580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:01.711541 12586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:01.711725 12580 server_base.cc:1034] running on GCE node
W20260928 02:49:01.711566 12588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:01.711519 12585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:01.712507 12580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:01.713670 12580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:01.714845 12580 hybrid_clock.cc:648] HybridClock initialized: now 1790563741714816 us; error 44 us; skew 500 ppm
Sep 28 02:49:01 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:01.717839 12580 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:49:01.718782 12580 webserver.cc:460] Webserver started at http://127.28.248.190:36397/ using document root <none> and password file <none>
I20260928 02:49:01.719385 12580 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:01.719447 12580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:01.719753 12580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:49:01.721894 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "3924dd43804d4dfd829f284297958fbd"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "67962d569c9fb8e8dc525df161e14b76"
server_key_iv: "6e3436106fcaed1fdcb65918dac16650"
server_key_version: "encryptionkey@0"
I20260928 02:49:01.722472 12580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "3924dd43804d4dfd829f284297958fbd"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "67962d569c9fb8e8dc525df161e14b76"
server_key_iv: "6e3436106fcaed1fdcb65918dac16650"
server_key_version: "encryptionkey@0"
I20260928 02:49:01.726509 12580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:49:01.729151 12595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:01.730406 12580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:49:01.730583 12580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "3924dd43804d4dfd829f284297958fbd"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "67962d569c9fb8e8dc525df161e14b76"
server_key_iv: "6e3436106fcaed1fdcb65918dac16650"
server_key_version: "encryptionkey@0"
I20260928 02:49:01.730683 12580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:01.744747 12580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:01.748071 12580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:01.748281 12580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:01.755421 12580 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:36499
I20260928 02:49:01.755812 12647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:36499 every 8 connection(s)
I20260928 02:49:01.756458 12580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:49:01.759336 12648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:01.765017 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap starting.
I20260928 02:49:01.766206 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12580
I20260928 02:49:01.766352 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:49:01.766654 29666 external_mini_cluster.cc:1428] Setting key 4dbc077cb6b592c2f67877db4bcb615c
I20260928 02:49:01.768071 12648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:01.768896 12648 log.cc:824] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:01.770754 12648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: No bootstrap required, opened a new log
I20260928 02:49:01.774299 12648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:01.774535 12648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:01.774626 12648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3924dd43804d4dfd829f284297958fbd, State: Initialized, Role: FOLLOWER
I20260928 02:49:01.775158 12648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:01.775324 12648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:49:01.775388 12648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 28 02:49:01 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:49:01.775490 12648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:01.776636 12648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:01.777030 12648 leader_election.cc:302] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3924dd43804d4dfd829f284297958fbd; no voters:
I20260928 02:49:01.777410 12648 leader_election.cc:288] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:49:01.777575 12653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:49:01.777797 12653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 LEADER]: Becoming Leader. State: Replica: 3924dd43804d4dfd829f284297958fbd, State: Running, Role: LEADER
I20260928 02:49:01.778137 12653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:01.778676 12648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:49:01.780176 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3924dd43804d4dfd829f284297958fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } } }
I20260928 02:49:01.780319 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:49:01.780176 12655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3924dd43804d4dfd829f284297958fbd. Latest consensus state: current_term: 1 leader_uuid: "3924dd43804d4dfd829f284297958fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } } }
I20260928 02:49:01.780673 12655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:49:01.780717 12662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:49:01.782080 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:01.767837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.28.248.190:36499)
0928 02:49:01.768246 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:01.768261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:01.768296 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:49:01.769044 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:01.769070 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:01.769415 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:01.769755 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:01.770535 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:01.771577 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:01.772561 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:01.773123 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:01.773304 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:01.776032 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:01.776038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:01.776055 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:01.776085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:01.778100 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:01.778620 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:01.778626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:01.778632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:01.778774 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:01.779069 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:01.779072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:01.779074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:01.779612 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:01.780481 (+ 869us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:01.780684 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:49:01.781035 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":145,"threads_started":1}
I20260928 02:49:01.783895 12662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:49:01.789139 12662 catalog_manager.cc:1347] Generated new cluster ID: 904246e10a5f47ccbfaecd43e2dcf11b
I20260928 02:49:01.789206 12662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:49:01.798497 12662 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:49:01.799760 12662 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:49:01.805635 12662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Generated new TSK 0
I20260928 02:49:01.806413 12662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:49:01.844210 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36499
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:49:01.962714 12674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:01.963003 12674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:01.963052 12674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:01.968040 12674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:01.968122 12674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:01.968189 12674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:49:01.974349 12674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:01.975428 12674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:01.976464 12674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:01.987112 12682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:01.987112 12679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:01.987246 12674 server_base.cc:1034] running on GCE node
W20260928 02:49:01.987453 12680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:01.987900 12674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:01.988703 12674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:01.989909 12674 hybrid_clock.cc:648] HybridClock initialized: now 1790563741989875 us; error 50 us; skew 500 ppm
Sep 28 02:49:01 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:01.992918 12674 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:49:01.993932 12674 webserver.cc:460] Webserver started at http://127.28.248.129:36675/ using document root <none> and password file <none>
I20260928 02:49:01.994549 12674 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:01.994647 12674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:01.994897 12674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:49:01.997193 12674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "4aec09e3e9554ab991cac5aa8b525b72"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "587d29a648481f4422c3750e1d0b9aa0"
server_key_iv: "a931cc839140180252992b789faddbd4"
server_key_version: "encryptionkey@0"
I20260928 02:49:01.997790 12674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "4aec09e3e9554ab991cac5aa8b525b72"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "587d29a648481f4422c3750e1d0b9aa0"
server_key_iv: "a931cc839140180252992b789faddbd4"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.001827 12674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:49:02.004410 12689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.005716 12674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:49:02.005867 12674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "4aec09e3e9554ab991cac5aa8b525b72"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "587d29a648481f4422c3750e1d0b9aa0"
server_key_iv: "a931cc839140180252992b789faddbd4"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.005988 12674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:02.028832 12674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:02.032569 12674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:02.032852 12674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:02.033486 12674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:02.034466 12674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:49:02.034538 12674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.034612 12674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:02.034665 12674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.044215 12674 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:37797
I20260928 02:49:02.044252 12802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:37797 every 8 connection(s)
I20260928 02:49:02.045197 12674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:49:02.051276 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12674
I20260928 02:49:02.051445 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:49:02.051725 29666 external_mini_cluster.cc:1428] Setting key 7257038c6262356e08e95f243721b08a
Sep 28 02:49:02 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:49:02.060906 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.046975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:38627 (local address 127.28.248.190:36499)
0928 02:49:02.047139 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.047143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.048189 (+ 1046us) server_negotiation.cc:407] Connection header received
0928 02:49:02.049243 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.049246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.049304 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.049421 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:02.050665 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.051265 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.052786 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.053136 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.053259 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.056489 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:02.056496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:02.056503 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:02.056538 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:02.058432 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.059090 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.059097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.059100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.059208 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.059556 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.059562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.059564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.059861 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:02.059981 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.060480 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.060678 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:49:02.061669 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.047362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:38627)
0928 02:49:02.048027 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.048070 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.049024 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.049615 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.049622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.049986 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:02.050473 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.050486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.051398 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.051405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.052663 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.052674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.053324 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.053329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.053580 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.054231 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:02.054257 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:02.056296 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:02.058684 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:02.058689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:02.058702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:02.058964 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:02.059315 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:02.059318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:02.059319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:02.059453 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:02.059983 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:02.059988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:02.060228 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.060382 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.060732 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:49:02.063006 12803 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36499
I20260928 02:49:02.063268 12803 heartbeater.cc:464] Registering TS with master...
I20260928 02:49:02.063800 12803 heartbeater.cc:511] Master 127.28.248.190:36499 requested a full tablet report, sending...
I20260928 02:49:02.065366 12612 ts_manager.cc:194] Registered new tserver with Master: 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:49:02.066608 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:38627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:49:02.083498 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36499
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:49:02.194053 12808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:02.194303 12808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:02.194355 12808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:02.198303 12808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:02.198361 12808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:02.198426 12808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:49:02.203549 12808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:02.204480 12808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:02.205503 12808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:02.215093 12813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:02.215122 12814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:02.215032 12816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:02.215345 12808 server_base.cc:1034] running on GCE node
I20260928 02:49:02.215747 12808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:02.216377 12808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:02.217541 12808 hybrid_clock.cc:648] HybridClock initialized: now 1790563742217518 us; error 38 us; skew 500 ppm
Sep 28 02:49:02 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563742, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:02.220260 12808 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:49:02.221104 12808 webserver.cc:460] Webserver started at http://127.28.248.130:37847/ using document root <none> and password file <none>
I20260928 02:49:02.221624 12808 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:02.221683 12808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:02.221917 12808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:49:02.223827 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "48e9816a480742f2a03b31d175060094"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "1b30d7835059a8dca36c0fbeb8f719e8"
server_key_iv: "49e0e534d5fcdf598b669d974385c999"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.224366 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "48e9816a480742f2a03b31d175060094"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "1b30d7835059a8dca36c0fbeb8f719e8"
server_key_iv: "49e0e534d5fcdf598b669d974385c999"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.227962 12808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:49:02.230330 12823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.231564 12808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:49:02.231712 12808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "48e9816a480742f2a03b31d175060094"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "1b30d7835059a8dca36c0fbeb8f719e8"
server_key_iv: "49e0e534d5fcdf598b669d974385c999"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.231827 12808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:02.243382 12808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:02.246498 12808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:02.246680 12808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:02.247221 12808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:02.247989 12808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:49:02.248059 12808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.248128 12808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:02.248183 12808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.256871 12808 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:39083
I20260928 02:49:02.256896 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:39083 every 8 connection(s)
I20260928 02:49:02.257714 12808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:49:02.259796 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12808
I20260928 02:49:02.259927 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:49:02.260250 29666 external_mini_cluster.cc:1428] Setting key 311afda97a7382f68946259492dd33c2
Sep 28 02:49:02 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563742, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:49:02.274199 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.259430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:52953 (local address 127.28.248.190:36499)
0928 02:49:02.259601 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.259606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.261124 (+ 1518us) server_negotiation.cc:407] Connection header received
0928 02:49:02.262116 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.262118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.262160 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.262225 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:02.263873 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.264796 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.266669 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.266919 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.267002 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.270012 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:02.270015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:02.270017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:02.270041 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:02.271714 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.272442 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.272447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.272451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.272558 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.272895 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.272897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.272898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.273144 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:02.273270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.273871 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.273981 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:49:02.275059 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.259883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:52953)
0928 02:49:02.260689 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.260743 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.261866 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.262421 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.262433 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.262968 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:02.263684 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.263707 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.264933 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.264936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.266498 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.266513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.267082 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.267088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.267330 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.268003 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:02.268033 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:02.269852 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:02.271860 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:02.271869 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:02.271883 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:02.272329 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:02.272669 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:02.272672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:02.272674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:02.272812 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:02.273270 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:02.273278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:02.273570 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.273807 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.274162 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":126,"threads_started":1}
I20260928 02:49:02.276247 12937 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36499
I20260928 02:49:02.276508 12937 heartbeater.cc:464] Registering TS with master...
I20260928 02:49:02.277035 12937 heartbeater.cc:511] Master 127.28.248.190:36499 requested a full tablet report, sending...
I20260928 02:49:02.278115 12612 ts_manager.cc:194] Registered new tserver with Master: 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083)
I20260928 02:49:02.279083 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:52953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:49:02.289505 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36499
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:49:02.397852 12942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:02.398126 12942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:02.398203 12942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:02.402170 12942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:02.402258 12942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:02.402349 12942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:49:02.407341 12942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:02.408294 12942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:02.409256 12942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:02.419019 12947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:02.419037 12950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:02.419127 12942 server_base.cc:1034] running on GCE node
W20260928 02:49:02.419013 12948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:02.419726 12942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:02.420369 12942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:02.421522 12942 hybrid_clock.cc:648] HybridClock initialized: now 1790563742421493 us; error 42 us; skew 500 ppm
Sep 28 02:49:02 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563742, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:02.424325 12942 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:49:02.425172 12942 webserver.cc:460] Webserver started at http://127.28.248.131:40255/ using document root <none> and password file <none>
I20260928 02:49:02.425695 12942 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:02.425752 12942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:02.426014 12942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:49:02.427884 12942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "54b957f939f842edb7efc58b15598287"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "7cf2875b94cd6767f98a3d27a4b46c18"
server_key_iv: "240d757252c80f36b18a4bcf91d3d0fa"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.428367 12942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "54b957f939f842edb7efc58b15598287"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "7cf2875b94cd6767f98a3d27a4b46c18"
server_key_iv: "240d757252c80f36b18a4bcf91d3d0fa"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.431862 12942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 02:49:02.434157 12957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.435447 12942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:49:02.435542 12942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "54b957f939f842edb7efc58b15598287"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "7cf2875b94cd6767f98a3d27a4b46c18"
server_key_iv: "240d757252c80f36b18a4bcf91d3d0fa"
server_key_version: "encryptionkey@0"
I20260928 02:49:02.435657 12942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:02.450709 12942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:02.453539 12942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:02.453732 12942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:02.454286 12942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:02.455140 12942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:49:02.455236 12942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.455307 12942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:02.455338 12942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:02.464349 12942 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:43933
I20260928 02:49:02.464371 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:43933 every 8 connection(s)
I20260928 02:49:02.465273 12942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:49:02 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563742, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:49:02.475008 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 12942
I20260928 02:49:02.475188 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:49:02.475484 29666 external_mini_cluster.cc:1428] Setting key 56d8ad71bee74d4dd3a0170d8e9e4632
I20260928 02:49:02.478695 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.466985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:49927 (local address 127.28.248.190:36499)
0928 02:49:02.467148 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.467153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.468203 (+ 1050us) server_negotiation.cc:407] Connection header received
0928 02:49:02.469069 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.469072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.469120 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.469193 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:02.470271 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.470801 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.471910 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.472127 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.472205 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.474811 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:02.474817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:02.474820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:02.474850 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:02.476527 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.477127 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.477131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.477133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.477222 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.477563 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.477566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.477568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.477757 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:02.477864 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.478403 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.478518 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:49:02.479516 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.467380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:49927)
0928 02:49:02.468061 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.468101 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.468894 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.469334 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.469340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.469684 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:02.470150 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.470163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.470906 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.470909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.471805 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.471814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.472252 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.472254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.472389 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.473005 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:02.473020 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:02.474614 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:02.476728 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:02.476734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:02.476749 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:02.477012 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:02.477318 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:02.477322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:02.477324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:02.477463 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:02.477861 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:02.477867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:02.478138 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.478347 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.478663 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":159,"threads_started":1}
I20260928 02:49:02.480665 13071 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36499
I20260928 02:49:02.480921 13071 heartbeater.cc:464] Registering TS with master...
I20260928 02:49:02.481391 13071 heartbeater.cc:511] Master 127.28.248.190:36499 requested a full tablet report, sending...
I20260928 02:49:02.482334 12612 ts_manager.cc:194] Registered new tserver with Master: 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
I20260928 02:49:02.482854 12612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:49927
I20260928 02:49:02.489122 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:49:02.498397 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.490937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.28.248.190:36499)
0928 02:49:02.491061 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.491065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.491320 (+ 255us) server_negotiation.cc:407] Connection header received
0928 02:49:02.491473 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.491476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.491526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.491617 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:02.492317 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.492995 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.493823 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.494064 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.494176 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.495452 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:02.495456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:02.495458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:02.495482 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:02.496737 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.497146 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.497151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.497153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.497240 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:02.497520 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:02.497523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:02.497524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:02.497738 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:02.497878 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.498126 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.498236 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:49:02.501189 12612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:49:02.504164 12612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:49:02.521133 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.514651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:37797 (local address 127.0.0.1:53206)
0928 02:49:02.515753 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.515768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.515856 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.516553 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.516558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.516577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:02.516741 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.516746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.517862 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.517867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.519250 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.519261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.520553 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.520556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.520685 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.520727 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.520837 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.520968 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":109,"threads_started":1}
I20260928 02:49:02.521566 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.515294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.28.248.129:37797)
0928 02:49:02.515697 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.515712 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.515866 (+ 154us) server_negotiation.cc:407] Connection header received
0928 02:49:02.516018 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.516024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.516239 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.516394 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:02.516855 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.517717 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.519391 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.520253 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.520362 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.520492 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.521315 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.521426 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
I20260928 02:49:02.522027 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.514483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:43933 (local address 127.0.0.1:53330)
0928 02:49:02.515370 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.515422 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.515545 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.516786 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.516790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.516808 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:02.516917 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.516938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.518827 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.518834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.520245 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.520256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.521587 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.521590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.521690 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.521713 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.521761 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.521830 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":28,"thread_start_us":96,"threads_started":1}
I20260928 02:49:02.522429 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.515656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:39083 (local address 127.0.0.1:47136)
0928 02:49:02.515850 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.515862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.515933 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.517018 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.517022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.517039 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:02.517130 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.517136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.518800 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.518803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.519977 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.519989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.522043 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.522046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.522134 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.522154 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.522236 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.522306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
I20260928 02:49:02.522488 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.516095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.28.248.130:39083)
0928 02:49:02.516556 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.516567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.516588 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:49:02.516689 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.516697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.516868 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.517268 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:02.517426 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.518665 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.520118 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.521939 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.522053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.522171 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.522262 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.522345 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260928 02:49:02.525166 12737 tablet_service.cc:1467] Processing CreateTablet for tablet bd4f5770e9bc43a2b3f977166b01f153 (DEFAULT_TABLE table=test-table [id=c745cb7441964bac834ce32f7ceb47c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:49:02.525573 12871 tablet_service.cc:1467] Processing CreateTablet for tablet bd4f5770e9bc43a2b3f977166b01f153 (DEFAULT_TABLE table=test-table [id=c745cb7441964bac834ce32f7ceb47c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:49:02.526500 12737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4f5770e9bc43a2b3f977166b01f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:02.526778 12871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4f5770e9bc43a2b3f977166b01f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:02.534535 13088 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap starting.
I20260928 02:49:02.534544 13089 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Bootstrap starting.
I20260928 02:49:02.535526 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.514660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.28.248.131:43933)
0928 02:49:02.516105 (+ 1445us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.516114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.516130 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:49:02.516190 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.516194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.516636 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.517077 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:02.517241 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.518660 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.520444 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.521439 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.534902 (+ 13463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.535092 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.535152 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.535263 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":124,"threads_started":1}
I20260928 02:49:02.537554 13089 tablet_bootstrap.cc:654] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:02.537710 13088 tablet_bootstrap.cc:654] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:02.538504 13089 log.cc:824] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:02.538924 13088 log.cc:824] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:02.538614 13005 tablet_service.cc:1467] Processing CreateTablet for tablet bd4f5770e9bc43a2b3f977166b01f153 (DEFAULT_TABLE table=test-table [id=c745cb7441964bac834ce32f7ceb47c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:49:02.539691 13005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4f5770e9bc43a2b3f977166b01f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:02.540830 13089 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: No bootstrap required, opened a new log
I20260928 02:49:02.541024 13089 ts_tablet_manager.cc:1397] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:49:02.545292 13089 raft_consensus.cc:348] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.545516 13089 raft_consensus.cc:374] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:02.545550 13089 raft_consensus.cc:729] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e9816a480742f2a03b31d175060094, State: Initialized, Role: FOLLOWER
I20260928 02:49:02.545979 13089 consensus_queue.cc:260] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.547147 12937 heartbeater.cc:503] Master 127.28.248.190:36499 was elected leader, sending a full tablet report...
I20260928 02:49:02.547158 13089 ts_tablet_manager.cc:1428] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:49:02.549023 13088 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: No bootstrap required, opened a new log
I20260928 02:49:02.549232 13088 ts_tablet_manager.cc:1397] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20260928 02:49:02.553071 13088 raft_consensus.cc:348] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.553269 13088 raft_consensus.cc:374] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:02.553510 13088 raft_consensus.cc:729] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aec09e3e9554ab991cac5aa8b525b72, State: Initialized, Role: FOLLOWER
I20260928 02:49:02.554018 13088 consensus_queue.cc:260] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.554738 12803 heartbeater.cc:503] Master 127.28.248.190:36499 was elected leader, sending a full tablet report...
I20260928 02:49:02.554963 13088 ts_tablet_manager.cc:1428] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 02:49:02.555330 13093 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Bootstrap starting.
I20260928 02:49:02.558154 13093 tablet_bootstrap.cc:654] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:02.558866 13093 log.cc:824] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:02.560679 13093 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: No bootstrap required, opened a new log
I20260928 02:49:02.560881 13093 ts_tablet_manager.cc:1397] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260928 02:49:02.564080 13093 raft_consensus.cc:348] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.564291 13093 raft_consensus.cc:374] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:02.564363 13093 raft_consensus.cc:729] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b957f939f842edb7efc58b15598287, State: Initialized, Role: FOLLOWER
I20260928 02:49:02.564790 13093 consensus_queue.cc:260] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.565435 13071 heartbeater.cc:503] Master 127.28.248.190:36499 was elected leader, sending a full tablet report...
I20260928 02:49:02.565608 13093 ts_tablet_manager.cc:1428] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:49:02.667677 13096 raft_consensus.cc:482] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:02.667886 13096 raft_consensus.cc:504] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.668998 13096 leader_election.cc:288] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797), 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083)
I20260928 02:49:02.672654 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.669298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:37797 (local address 127.28.248.131:53547)
0928 02:49:02.669450 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.669486 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.669601 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.669864 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.669867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.669884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:02.670008 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.670013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.670761 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.670766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.671554 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.671563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.672196 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.672200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.672307 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.672335 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.672425 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.672496 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:49:02.672650 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.669427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:53547 (local address 127.28.248.129:37797)
0928 02:49:02.669574 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.669578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.669614 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:49:02.669698 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.669704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.669760 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.669855 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:02.670137 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.670647 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.671683 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.672076 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.672207 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.672261 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.672440 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.672509 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:49:02.673192 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.669656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:39083 (local address 127.28.248.131:35055)
0928 02:49:02.669989 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.670002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:02.670146 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:02.670415 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:02.670418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:02.670433 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:02.670555 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.670560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.671293 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.671299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.672055 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:02.672063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.672776 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.672780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:02.672885 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.672901 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:49:02.672988 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:49:02.673060 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260928 02:49:02.673179 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.669783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:35055 (local address 127.28.248.130:39083)
0928 02:49:02.669992 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.669998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.670106 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:49:02.670262 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.670265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.670321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.670407 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:02.670679 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.671182 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.672184 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.672592 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.672765 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.672813 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.673004 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.673073 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 02:49:02.673527 12757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4f5770e9bc43a2b3f977166b01f153" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aec09e3e9554ab991cac5aa8b525b72" is_pre_election: true
I20260928 02:49:02.673873 12757 raft_consensus.cc:2454] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 0.
I20260928 02:49:02.673902 12891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4f5770e9bc43a2b3f977166b01f153" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e9816a480742f2a03b31d175060094" is_pre_election: true
I20260928 02:49:02.674196 12891 raft_consensus.cc:2454] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 0.
I20260928 02:49:02.674350 12958 leader_election.cc:302] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72, 54b957f939f842edb7efc58b15598287; no voters:
I20260928 02:49:02.674615 13096 raft_consensus.cc:2781] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:49:02.674691 13096 raft_consensus.cc:482] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:49:02.674723 13096 raft_consensus.cc:3037] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:02.675740 13096 raft_consensus.cc:504] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.676070 13096 leader_election.cc:288] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 election: Requested vote from peers 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797), 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083)
I20260928 02:49:02.676479 12757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4f5770e9bc43a2b3f977166b01f153" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aec09e3e9554ab991cac5aa8b525b72"
I20260928 02:49:02.676635 12757 raft_consensus.cc:3037] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:02.676679 12891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4f5770e9bc43a2b3f977166b01f153" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e9816a480742f2a03b31d175060094"
I20260928 02:49:02.676780 12891 raft_consensus.cc:3037] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:02.677944 12757 raft_consensus.cc:2454] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 1.
I20260928 02:49:02.677944 12891 raft_consensus.cc:2454] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 1.
I20260928 02:49:02.678253 12958 leader_election.cc:302] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72, 54b957f939f842edb7efc58b15598287; no voters:
I20260928 02:49:02.678496 13096 raft_consensus.cc:2781] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:49:02.678754 13096 raft_consensus.cc:686] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 1 LEADER]: Becoming Leader. State: Replica: 54b957f939f842edb7efc58b15598287, State: Running, Role: LEADER
I20260928 02:49:02.679090 13096 consensus_queue.cc:237] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:02.682327 12612 catalog_manager.cc:5495] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54b957f939f842edb7efc58b15598287 (127.28.248.131). New cstate: current_term: 1 leader_uuid: "54b957f939f842edb7efc58b15598287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } health_report { overall_health: UNKNOWN } } }
W20260928 02:49:02.716172 13072 tablet.cc:2369] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:49:02.719118 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.715680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.28.248.131:43933)
0928 02:49:02.715806 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:02.715810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:02.715856 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:49:02.715935 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:02.715937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:02.716011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:02.716102 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:49:02.716447 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.716974 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.717806 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:02.718019 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:02.718113 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:02.718381 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:49:02.718799 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:49:02.718893 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:02.718944 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:49:02.718996 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:49:02.725243 12757 raft_consensus.cc:1261] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Refusing update from remote peer 54b957f939f842edb7efc58b15598287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:49:02.725243 12891 raft_consensus.cc:1261] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Refusing update from remote peer 54b957f939f842edb7efc58b15598287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:49:02.725950 13096 consensus_queue.cc:1035] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:49:02.726126 13098 consensus_queue.cc:1035] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:49:02.732740 13101 mvcc.cc:205] Tried to move back new op lower bound from 7334149090195181568 to 7334149090016202752. Current Snapshot: MvccSnapshot[applied={T|T < 7334149090195181568}]
I20260928 02:49:02.735776 13102 mvcc.cc:205] Tried to move back new op lower bound from 7334149090195181568 to 7334149090016202752. Current Snapshot: MvccSnapshot[applied={T|T < 7334149090195181568}]
I20260928 02:49:02.740746 12611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:49:02.740913 12611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:49:02.743234 12611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Sending DeleteTablet for 3 replicas of tablet bd4f5770e9bc43a2b3f977166b01f153
I20260928 02:49:02.743952 13005 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4f5770e9bc43a2b3f977166b01f153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:49:02 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:53330
I20260928 02:49:02.744051 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12674
I20260928 02:49:02.744109 12871 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4f5770e9bc43a2b3f977166b01f153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:49:02 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:47136
I20260928 02:49:02.744851 13111 tablet_replica.cc:330] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: stopping tablet replica
I20260928 02:49:02.745133 13111 raft_consensus.cc:2229] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:49:02.745431 13111 raft_consensus.cc:2258] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:49:02.746567 13111 ts_tablet_manager.cc:1905] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:49:02.748870 13111 ts_tablet_manager.cc:1918] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:49:02.748963 13111 log.cc:1196] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/bd4f5770e9bc43a2b3f977166b01f153
I20260928 02:49:02.749266 13111 ts_tablet_manager.cc:1939] T bd4f5770e9bc43a2b3f977166b01f153 P 54b957f939f842edb7efc58b15598287: Deleting consensus metadata
I20260928 02:49:02.750151 13112 tablet_replica.cc:330] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: stopping tablet replica
I20260928 02:49:02.750277 12597 catalog_manager.cc:4841] TS 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): tablet bd4f5770e9bc43a2b3f977166b01f153 (table test-table [id=c745cb7441964bac834ce32f7ceb47c2]) successfully deleted
I20260928 02:49:02.750506 13112 raft_consensus.cc:2229] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:49:02.750804 13112 raft_consensus.cc:2258] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:49:02.751986 13112 ts_tablet_manager.cc:1905] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:49:02.753280 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12808
W20260928 02:49:02.754063 12596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:37797 (error 108)
I20260928 02:49:02.755023 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.754759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:37797 (local address 127.0.0.1:53218)
0928 02:49:02.754886 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.754953 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:49:02.755158 12596 catalog_manager.cc:4584] TS 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd4f5770e9bc43a2b3f977166b01f153: Network error: Client connection negotiation failed: client connection to 127.28.248.129:37797: connect: Connection refused (error 111)
I20260928 02:49:02.761176 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:02.760912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:39083 (local address 127.0.0.1:47148)
0928 02:49:02.761005 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:02.761064 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.130:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 02:49:02.761787 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12942
I20260928 02:49:02.767138 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 12580
I20260928 02:49:22.774000 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36499
--webserver_interface=127.28.248.190
--webserver_port=36397
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:36499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:49:22.883351 13114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:22.883656 13114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:22.883729 13114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:22.887836 13114 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:49:22.887919 13114 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:49:22.887949 13114 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:49:22.887984 13114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:49:22.888036 13114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:22.888077 13114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:22.888113 13114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:49:22.888146 13114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:49:22.893638 13114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:36499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=36397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:22.894771 13114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:22.895833 13114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:22.909775 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:22.909914 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:22.909873 13114 server_base.cc:1034] running on GCE node
W20260928 02:49:22.909780 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:22.910787 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:22.912030 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:22.913194 13114 hybrid_clock.cc:648] HybridClock initialized: now 1790563762913159 us; error 52 us; skew 500 ppm
Sep 28 02:49:22 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563762, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:22.916236 13114 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:49:22.917346 13114 webserver.cc:460] Webserver started at http://127.28.248.190:36397/ using document root <none> and password file <none>
I20260928 02:49:22.918052 13114 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:22.918167 13114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:22.922678 13114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:49:22.925091 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:22.926407 13114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:49:22.926612 13114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "3924dd43804d4dfd829f284297958fbd"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "67962d569c9fb8e8dc525df161e14b76"
server_key_iv: "6e3436106fcaed1fdcb65918dac16650"
server_key_version: "encryptionkey@0"
I20260928 02:49:22.927222 13114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:22.939648 13114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:22.942905 13114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:22.943135 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:22.950899 13114 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:36499
I20260928 02:49:22.951229 13182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:36499 every 8 connection(s)
I20260928 02:49:22.952064 13114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:49:22.956554 13183 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:49:22.958213 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap starting.
I20260928 02:49:22.960911 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13114
I20260928 02:49:22.961354 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:37797
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:36499
--webserver_port=36675
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:49:22.975345 13183 log.cc:824] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:22.984007 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:49:22.984370 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap complete.
I20260928 02:49:22.988530 13183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:22.989069 13183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3924dd43804d4dfd829f284297958fbd, State: Initialized, Role: FOLLOWER
I20260928 02:49:22.989553 13183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:22.989693 13183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:49:22.989782 13183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:49:22.989897 13183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:49:22.991925 13183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:22.992254 13183 leader_election.cc:302] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3924dd43804d4dfd829f284297958fbd; no voters:
I20260928 02:49:22.992555 13183 leader_election.cc:288] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:49:22.992733 13187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:49:22.992985 13187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 LEADER]: Becoming Leader. State: Replica: 3924dd43804d4dfd829f284297958fbd, State: Running, Role: LEADER
I20260928 02:49:22.993290 13187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:22.993731 13183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:49:22.994283 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3924dd43804d4dfd829f284297958fbd. Latest consensus state: current_term: 2 leader_uuid: "3924dd43804d4dfd829f284297958fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } } }
I20260928 02:49:22.994397 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:49:22.994670 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3924dd43804d4dfd829f284297958fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } } }
I20260928 02:49:22.994791 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:49:22.998445 13202 catalog_manager.cc:1259] Loaded cluster ID: 904246e10a5f47ccbfaecd43e2dcf11b
I20260928 02:49:22.998520 13202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: loading cluster ID for follower catalog manager: success
I20260928 02:49:23.000202 13202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: acquiring CA information for follower catalog manager: success
I20260928 02:49:23.001421 13202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:49:23.001566 13192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:49:23.003463 13192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:49:23.003890 13192 catalog_manager.cc:1259] Loaded cluster ID: 904246e10a5f47ccbfaecd43e2dcf11b
I20260928 02:49:23.003960 13192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:49:23.004506 13192 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:49:23.004956 13192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Loaded TSK: 0
I20260928 02:49:23.027096 13192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Generated new TSK 1
I20260928 02:49:23.027732 13192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:49:23.091421 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:23.091682 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:23.091755 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:23.096180 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:23.096278 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:23.096377 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:49:23.101689 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:37797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=36675
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:23.102718 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:23.103725 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:23.114238 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:23.114298 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:23.114246 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:23.115068 13185 server_base.cc:1034] running on GCE node
I20260928 02:49:23.115456 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:23.116123 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:23.117312 13185 hybrid_clock.cc:648] HybridClock initialized: now 1790563763117288 us; error 41 us; skew 500 ppm
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563763, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:23.120217 13185 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:49:23.121232 13185 webserver.cc:460] Webserver started at http://127.28.248.129:36675/ using document root <none> and password file <none>
I20260928 02:49:23.121799 13185 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:23.121871 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:23.125815 13185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:49:23.128053 13220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:23.129417 13185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:49:23.129515 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "4aec09e3e9554ab991cac5aa8b525b72"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "587d29a648481f4422c3750e1d0b9aa0"
server_key_iv: "a931cc839140180252992b789faddbd4"
server_key_version: "encryptionkey@0"
I20260928 02:49:23.130136 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:23.154620 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:23.158140 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:23.158361 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:23.159052 13185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:23.160200 13227 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:49:23.164359 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:49:23.164459 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 02:49:23.164544 13185 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:49:23.166304 13185 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:49:23.166383 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:49:23.166669 13227 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap starting.
I20260928 02:49:23.177174 13185 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:37797
I20260928 02:49:23.177246 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:37797 every 8 connection(s)
I20260928 02:49:23.178237 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:49:23.185333 13227 log.cc:824] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:23.188306 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13185
I20260928 02:49:23.188870 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:39083
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:36499
--webserver_port=37847
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563763, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:49:23.195230 13227 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:49:23.195703 13227 tablet_bootstrap.cc:492] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap complete.
I20260928 02:49:23.197000 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.180322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:58793)
0928 02:49:23.180826 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.180879 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.181702 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.183005 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.183012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.183358 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:23.183817 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.183831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.185619 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.185622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.186771 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.186782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.187775 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.187778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.187912 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.188676 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:23.188700 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:23.190866 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:23.193393 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.193402 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.193418 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.193828 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.194201 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.194206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.194208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.194447 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.195126 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:23.195132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:23.195423 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.195634 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.196070 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260928 02:49:23.196988 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.180353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:58793 (local address 127.28.248.190:36499)
0928 02:49:23.180908 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.180926 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.180978 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:49:23.182137 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.182152 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.182565 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.182984 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:23.183991 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.185448 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.186910 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.187585 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.187754 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.191073 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:23.191081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:23.191098 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:23.191125 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:23.193186 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.193954 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.193960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.193966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.194102 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.194545 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.194549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.194551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.195003 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:23.195153 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.195801 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.196090 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":159,"threads_started":1}
I20260928 02:49:23.197281 13227 ts_tablet_manager.cc:1397] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260928 02:49:23.199478 13335 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36499
I20260928 02:49:23.199716 13335 heartbeater.cc:464] Registering TS with master...
I20260928 02:49:23.200191 13335 heartbeater.cc:511] Master 127.28.248.190:36499 requested a full tablet report, sending...
I20260928 02:49:23.201174 13227 raft_consensus.cc:348] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:23.201828 13227 raft_consensus.cc:729] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aec09e3e9554ab991cac5aa8b525b72, State: Initialized, Role: FOLLOWER
I20260928 02:49:23.202373 13227 consensus_queue.cc:260] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } } peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } }
I20260928 02:49:23.202732 13147 ts_manager.cc:194] Registered new tserver with Master: 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797)
I20260928 02:49:23.203387 13227 ts_tablet_manager.cc:1428] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:49:23.206395 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:58793
I20260928 02:49:23.208834 13335 heartbeater.cc:503] Master 127.28.248.190:36499 was elected leader, sending a full tablet report...
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563762, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.129@KRBTEST.COM
I20260928 02:49:23.217520 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.205673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:37797 (local address 127.0.0.1:35014)
0928 02:49:23.206242 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.206289 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.206426 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.206960 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.206966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.207405 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:23.207536 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.207545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.209151 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.209154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.210109 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.210120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.210896 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.210899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.210978 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.211660 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:23.211672 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:23.213333 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:23.215646 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.215654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.215659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.215940 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.216284 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.216287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.216289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.216376 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.216943 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:23.216948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:23.217083 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.217207 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.217382 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":110,"threads_started":1}
I20260928 02:49:23.217669 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.205763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.28.248.129:37797)
0928 02:49:23.206374 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.206385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.206411 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:49:23.206610 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.206616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.206811 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.206938 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:23.207678 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.209027 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.210237 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.210773 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.210882 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.213482 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:23.213494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:23.213512 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:23.213542 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:23.215506 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.216078 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.216084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.216085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.216181 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.216472 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.216477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.216478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.216821 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:23.216950 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.217312 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.217489 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":518,"thread_start_us":97,"threads_started":1}
I20260928 02:49:23.218691 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4f5770e9bc43a2b3f977166b01f153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:49:02 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:35014
I20260928 02:49:23.219587 13344 tablet_replica.cc:330] stopping tablet replica
I20260928 02:49:23.219898 13344 raft_consensus.cc:2229] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:49:23.220161 13344 raft_consensus.cc:2258] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:49:23.221187 13344 ts_tablet_manager.cc:1905] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:49:23.224949 13344 ts_tablet_manager.cc:1918] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:49:23.225035 13344 log.cc:1196] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal/wals/bd4f5770e9bc43a2b3f977166b01f153
I20260928 02:49:23.225337 13344 ts_tablet_manager.cc:1939] T bd4f5770e9bc43a2b3f977166b01f153 P 4aec09e3e9554ab991cac5aa8b525b72: Deleting consensus metadata
I20260928 02:49:23.227278 13131 catalog_manager.cc:4841] TS 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797): tablet bd4f5770e9bc43a2b3f977166b01f153 (table test-table [id=c745cb7441964bac834ce32f7ceb47c2]) successfully deleted
W20260928 02:49:23.309881 13340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:23.310129 13340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:23.310173 13340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:23.314159 13340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:23.314213 13340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:23.314270 13340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:49:23.319585 13340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:39083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=37847
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:23.320547 13340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:23.321619 13340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:23.331195 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:23.331213 13352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:23.331282 13340 server_base.cc:1034] running on GCE node
W20260928 02:49:23.331426 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:23.331936 13340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:23.332556 13340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:23.333810 13340 hybrid_clock.cc:648] HybridClock initialized: now 1790563763333790 us; error 27 us; skew 500 ppm
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563763, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:23.336683 13340 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:49:23.337539 13340 webserver.cc:460] Webserver started at http://127.28.248.130:37847/ using document root <none> and password file <none>
I20260928 02:49:23.338126 13340 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:23.338243 13340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:23.342005 13340 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 02:49:23.344106 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:23.345381 13340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:49:23.345512 13340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "48e9816a480742f2a03b31d175060094"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "1b30d7835059a8dca36c0fbeb8f719e8"
server_key_iv: "49e0e534d5fcdf598b669d974385c999"
server_key_version: "encryptionkey@0"
I20260928 02:49:23.346048 13340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:23.361770 13340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:23.364792 13340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:23.364989 13340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:23.365576 13340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:23.366673 13366 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 02:49:23.370390 13366 ts_tablet_manager.cc:1852] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 02:49:23.370786 13366 ts_tablet_manager.cc:1905] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:49:23.374919 13366 ts_tablet_manager.cc:1918] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:49:23.375037 13366 log.cc:1196] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/bd4f5770e9bc43a2b3f977166b01f153
I20260928 02:49:23.375346 13366 ts_tablet_manager.cc:1939] T bd4f5770e9bc43a2b3f977166b01f153 P 48e9816a480742f2a03b31d175060094: Deleting consensus metadata
I20260928 02:49:23.375680 13340 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 02:49:23.375769 13340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260928 02:49:23.375862 13340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:23.375895 13340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:23.385456 13340 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:39083
I20260928 02:49:23.385540 13473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:39083 every 8 connection(s)
I20260928 02:49:23.386415 13340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:49:23.386790 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13340
I20260928 02:49:23.387321 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:43933
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:36499
--webserver_port=40255
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563763, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:49:23.401613 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.388443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:46013 (local address 127.28.248.190:36499)
0928 02:49:23.388685 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.388692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.389685 (+ 993us) server_negotiation.cc:407] Connection header received
0928 02:49:23.390763 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.390770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.390834 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.390943 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:23.392214 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.393176 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.394500 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.394978 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.395120 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.397932 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:23.397938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:23.397943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:23.397975 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:23.399623 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.400107 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.400112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.400114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.400201 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.400521 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.400524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.400526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.400756 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:23.400850 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.401311 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.401439 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 02:49:23.402375 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.389017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:46013)
0928 02:49:23.389501 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.389554 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.390544 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.391110 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.391117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.391554 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:23.392047 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.392061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.393326 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.393329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.394381 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.394390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.395139 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.395142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.395330 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.395965 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:23.395982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:23.397760 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:23.399739 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.399743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.399754 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.400016 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.400297 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.400300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.400301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.400434 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.400856 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:23.400861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:23.401092 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.401247 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.401484 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260928 02:49:23.403529 13474 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36499
I20260928 02:49:23.403776 13474 heartbeater.cc:464] Registering TS with master...
I20260928 02:49:23.404299 13474 heartbeater.cc:511] Master 127.28.248.190:36499 requested a full tablet report, sending...
I20260928 02:49:23.405402 13147 ts_manager.cc:194] Registered new tserver with Master: 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083)
I20260928 02:49:23.406037 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:46013
W20260928 02:49:23.503034 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:23.503357 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:23.503473 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:23.507723 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:23.507812 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:23.507900 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:49:23.513450 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:43933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=40255
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:23.514544 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:23.515571 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:23.525451 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:23.525465 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:23.525449 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:23.525846 13476 server_base.cc:1034] running on GCE node
I20260928 02:49:23.526257 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:23.526970 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:23.528170 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790563763528153 us; error 28 us; skew 500 ppm
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563763, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:23.531131 13476 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:49:23.532119 13476 webserver.cc:460] Webserver started at http://127.28.248.131:40255/ using document root <none> and password file <none>
I20260928 02:49:23.532801 13476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:23.532905 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:23.538162 13476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:49:23.540431 13492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:23.541817 13476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:49:23.541913 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "54b957f939f842edb7efc58b15598287"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "7cf2875b94cd6767f98a3d27a4b46c18"
server_key_iv: "240d757252c80f36b18a4bcf91d3d0fa"
server_key_version: "encryptionkey@0"
I20260928 02:49:23.542469 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:23.560117 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:23.563374 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:23.563611 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:23.564306 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:23.565215 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:49:23.565263 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:23.565327 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:23.565359 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:23.574375 13476 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:43933
I20260928 02:49:23.574380 13605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:43933 every 8 connection(s)
I20260928 02:49:23.575441 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:49:23 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563763, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:49:23.584792 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13476
I20260928 02:49:23.589643 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.577435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:42053 (local address 127.28.248.190:36499)
0928 02:49:23.577592 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.577596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.578666 (+ 1070us) server_negotiation.cc:407] Connection header received
0928 02:49:23.579582 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.579585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.579634 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.579715 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:23.580979 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.581557 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.582648 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.582877 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.582960 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.585535 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:23.585539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:23.585542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:23.585578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:23.587322 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.587933 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.587935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.587937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.588008 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.588368 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.588371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.588373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.588653 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:23.588752 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.589333 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.589464 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:49:23.590592 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.577822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:42053)
0928 02:49:23.578503 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.578543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.579338 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.579881 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.579888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.580272 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:23.580834 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.580850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.581671 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.581674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.582531 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.582539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.583007 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.583010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.583166 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.583717 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:23.583733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:23.585334 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:23.587444 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.587450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.587464 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.587828 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.588098 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:23.588104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:23.588109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:23.588275 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:23.588790 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:23.588796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:23.589045 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.589242 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.589577 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":127,"threads_started":1}
I20260928 02:49:23.591881 13606 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36499
I20260928 02:49:23.592183 13606 heartbeater.cc:464] Registering TS with master...
I20260928 02:49:23.592736 13606 heartbeater.cc:511] Master 127.28.248.190:36499 requested a full tablet report, sending...
I20260928 02:49:23.593796 13147 ts_manager.cc:194] Registered new tserver with Master: 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
I20260928 02:49:23.594389 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:42053
I20260928 02:49:23.594739 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.585524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.28.248.190:36499)
0928 02:49:23.585795 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.585800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.586032 (+ 232us) server_negotiation.cc:407] Connection header received
0928 02:49:23.586173 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.586175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.586215 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.586296 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:23.587252 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.588073 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.589128 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.589490 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.589673 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.591283 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:23.591286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:23.591289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:23.591312 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:23.592920 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.593380 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.593383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.593384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.593458 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.593794 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.593797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.593799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.594014 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:23.594222 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.594404 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.594534 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260928 02:49:23.596428 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:49:23.600734 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.597268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.28.248.190:36499)
0928 02:49:23.597396 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.597401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.597547 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:49:23.597708 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.597711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.597809 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.597914 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:49:23.598168 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.598676 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.599535 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.599781 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.599891 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.600194 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:49:23.600321 (+ 127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:49:23.600579 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36424: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 02:49:23.600811 13609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36424: token expired
W20260928 02:49:23.601089 13077 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.248.190:36499)
I20260928 02:49:23.608439 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.601479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.28.248.190:36499)
0928 02:49:23.601585 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.601588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.601598 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:49:23.601662 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.601664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.601695 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.601753 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:23.602275 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.602774 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.603505 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.603751 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.603837 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.605305 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:23.605308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:23.605310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:23.605330 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:23.606846 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.607280 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.607282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.607283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.607361 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:23.607653 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:23.607655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:23.607656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:23.607811 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:23.607907 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.608130 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.608218 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:49:23.610868 13147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:49:23.614185 13147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:49:23.622237 13269 tablet_service.cc:1467] Processing CreateTablet for tablet f77bcf35a1f7408eb032978ba9c9b2dc (DEFAULT_TABLE table=test-table [id=605b994dd62b442f889e3c0677388158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:49:23.622931 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77bcf35a1f7408eb032978ba9c9b2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:23.625737 13227 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap starting.
I20260928 02:49:23.626454 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.621103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:39083 (local address 127.0.0.1:43380)
0928 02:49:23.621236 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.621251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.621331 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.622454 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.622458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.622474 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:23.622570 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.622579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.623698 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.623702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.624883 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.624894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.625972 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.625975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.626092 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.626150 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.626207 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.626284 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:49:23.626672 13227 tablet_bootstrap.cc:654] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:23.626776 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.621549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:43933 (local address 127.0.0.1:45320)
0928 02:49:23.621826 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.621840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.621924 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.622683 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.622685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.622702 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:23.622785 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.622790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.624070 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.624073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.625151 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.625158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.626199 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.626204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.626281 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.626300 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.626387 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.626443 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":127,"threads_started":1}
I20260928 02:49:23.627051 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.621302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.28.248.130:39083)
0928 02:49:23.621909 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.621922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.621944 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:49:23.622028 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.622035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.622272 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.622450 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:23.622677 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.623578 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.625012 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.625846 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.626604 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.626737 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.626785 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.626880 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":142,"threads_started":1}
I20260928 02:49:23.627103 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.621705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.28.248.131:43933)
0928 02:49:23.622142 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.622152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.622171 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:49:23.622236 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.622241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.622402 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.622529 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:23.622911 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.623951 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.625259 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.626065 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.626180 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.626304 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.626416 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.626757 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":125,"threads_started":1}
I20260928 02:49:23.628311 13227 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: No bootstrap required, opened a new log
I20260928 02:49:23.628424 13227 ts_tablet_manager.cc:1397] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:49:23.628414 13408 tablet_service.cc:1467] Processing CreateTablet for tablet f77bcf35a1f7408eb032978ba9c9b2dc (DEFAULT_TABLE table=test-table [id=605b994dd62b442f889e3c0677388158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:49:23.629082 13227 raft_consensus.cc:348] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.629241 13227 raft_consensus.cc:374] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:23.629289 13227 raft_consensus.cc:729] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aec09e3e9554ab991cac5aa8b525b72, State: Initialized, Role: FOLLOWER
I20260928 02:49:23.629410 13227 consensus_queue.cc:260] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.629495 13408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77bcf35a1f7408eb032978ba9c9b2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:23.629667 13227 ts_tablet_manager.cc:1428] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:49:23.630259 13540 tablet_service.cc:1467] Processing CreateTablet for tablet f77bcf35a1f7408eb032978ba9c9b2dc (DEFAULT_TABLE table=test-table [id=605b994dd62b442f889e3c0677388158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:49:23.631196 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77bcf35a1f7408eb032978ba9c9b2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:49:23.634342 13366 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Bootstrap starting.
I20260928 02:49:23.636372 13366 tablet_bootstrap.cc:654] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:23.637167 13366 log.cc:824] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:23.638212 13620 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Bootstrap starting.
I20260928 02:49:23.639032 13366 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: No bootstrap required, opened a new log
I20260928 02:49:23.639218 13366 ts_tablet_manager.cc:1397] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:49:23.640569 13620 tablet_bootstrap.cc:654] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Neither blocks nor log segments found. Creating new log.
I20260928 02:49:23.641261 13620 log.cc:824] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:23.642366 13366 raft_consensus.cc:348] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.642568 13366 raft_consensus.cc:374] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:23.642625 13366 raft_consensus.cc:729] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e9816a480742f2a03b31d175060094, State: Initialized, Role: FOLLOWER
I20260928 02:49:23.642948 13620 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: No bootstrap required, opened a new log
I20260928 02:49:23.643136 13620 ts_tablet_manager.cc:1397] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 02:49:23.643085 13366 consensus_queue.cc:260] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.643730 13474 heartbeater.cc:503] Master 127.28.248.190:36499 was elected leader, sending a full tablet report...
I20260928 02:49:23.643975 13366 ts_tablet_manager.cc:1428] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 02:49:23.646694 13620 raft_consensus.cc:348] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.646903 13620 raft_consensus.cc:374] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:49:23.646962 13620 raft_consensus.cc:729] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b957f939f842edb7efc58b15598287, State: Initialized, Role: FOLLOWER
I20260928 02:49:23.647385 13620 consensus_queue.cc:260] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.647981 13606 heartbeater.cc:503] Master 127.28.248.190:36499 was elected leader, sending a full tablet report...
I20260928 02:49:23.648278 13620 ts_tablet_manager.cc:1428] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260928 02:49:23.679560 13336 tablet.cc:2369] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:49:23.701910 13623 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:23.702157 13623 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.703394 13623 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797)
I20260928 02:49:23.707903 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.703691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:39083 (local address 127.28.248.131:42099)
0928 02:49:23.703845 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.703864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.704007 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.704340 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.704343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.704359 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:23.704483 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.704493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.705510 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.705513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.706675 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.706686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.707552 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.707554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.707634 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.707654 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.707717 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.707768 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:49:23.707996 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.703835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:42099 (local address 127.28.248.130:39083)
0928 02:49:23.704005 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.704009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.704028 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:49:23.704128 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.704134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.704194 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.704331 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:23.704657 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.705396 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.706809 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.707410 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.707534 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.707593 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.707761 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.707847 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:49:23.708381 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.703975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:37797 (local address 127.28.248.131:34311)
0928 02:49:23.704328 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.704339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:23.704480 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:23.704825 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:23.704828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:23.704843 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:49:23.704955 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.704960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.705902 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.705905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.706963 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:23.706970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.707933 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.707938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:23.708032 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.708060 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:49:23.708198 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 02:49:23.708259 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":127,"threads_started":1}
I20260928 02:49:23.708403 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.704128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:34311 (local address 127.28.248.129:37797)
0928 02:49:23.704264 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.704270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.704469 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:49:23.704584 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.704589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.704682 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.704790 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:49:23.705092 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.705785 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.707086 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.707787 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.707914 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.708072 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.708219 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.708294 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:49:23.708863 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77bcf35a1f7408eb032978ba9c9b2dc" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e9816a480742f2a03b31d175060094" is_pre_election: true
I20260928 02:49:23.708993 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77bcf35a1f7408eb032978ba9c9b2dc" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aec09e3e9554ab991cac5aa8b525b72" is_pre_election: true
I20260928 02:49:23.709228 13428 raft_consensus.cc:2454] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 0.
I20260928 02:49:23.709306 13289 raft_consensus.cc:2454] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 0.
I20260928 02:49:23.709772 13493 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e9816a480742f2a03b31d175060094, 54b957f939f842edb7efc58b15598287; no voters:
I20260928 02:49:23.710064 13623 raft_consensus.cc:2781] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:49:23.710161 13623 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:49:23.710212 13623 raft_consensus.cc:3037] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:23.711323 13623 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.711684 13623 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 election: Requested vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797)
I20260928 02:49:23.712033 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77bcf35a1f7408eb032978ba9c9b2dc" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e9816a480742f2a03b31d175060094"
I20260928 02:49:23.712034 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f77bcf35a1f7408eb032978ba9c9b2dc" candidate_uuid: "54b957f939f842edb7efc58b15598287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aec09e3e9554ab991cac5aa8b525b72"
I20260928 02:49:23.712152 13428 raft_consensus.cc:3037] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:23.712189 13289 raft_consensus.cc:3037] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:49:23.713320 13289 raft_consensus.cc:2454] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 1.
I20260928 02:49:23.713326 13428 raft_consensus.cc:2454] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54b957f939f842edb7efc58b15598287 in term 1.
I20260928 02:49:23.713617 13493 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72, 54b957f939f842edb7efc58b15598287; no voters:
I20260928 02:49:23.713791 13623 raft_consensus.cc:2781] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:49:23.714042 13623 raft_consensus.cc:686] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 1 LEADER]: Becoming Leader. State: Replica: 54b957f939f842edb7efc58b15598287, State: Running, Role: LEADER
I20260928 02:49:23.714394 13623 consensus_queue.cc:237] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:23.717597 13147 catalog_manager.cc:5495] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54b957f939f842edb7efc58b15598287 (127.28.248.131). New cstate: current_term: 1 leader_uuid: "54b957f939f842edb7efc58b15598287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } health_report { overall_health: UNKNOWN } } }
I20260928 02:49:23.732527 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.729149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.28.248.131:43933)
0928 02:49:23.729271 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:23.729275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:23.729287 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:49:23.729350 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:23.729352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:23.729402 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:23.729494 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:49:23.729717 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.730230 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.731097 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:23.731326 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:23.731479 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:23.731663 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:49:23.732153 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:49:23.732244 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:23.732348 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:49:23.732415 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:49:23.738834 13428 raft_consensus.cc:1261] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Refusing update from remote peer 54b957f939f842edb7efc58b15598287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:49:23.738834 13289 raft_consensus.cc:1261] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Refusing update from remote peer 54b957f939f842edb7efc58b15598287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:49:23.739607 13625 consensus_queue.cc:1035] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:49:23.739850 13623 consensus_queue.cc:1035] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:49:23.748363 13628 mvcc.cc:205] Tried to move back new op lower bound from 7334149176266682368 to 7334149176176652288. Current Snapshot: MvccSnapshot[applied={T|T < 7334149176266682368 or (T in {7334149176266682368})}]
I20260928 02:49:23.754971 13146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:49:23.755162 13146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:49:23.757577 13146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Sending DeleteTablet for 3 replicas of tablet f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:23.758309 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet f77bcf35a1f7408eb032978ba9c9b2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:49:23 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:45320
I20260928 02:49:23.758455 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet f77bcf35a1f7408eb032978ba9c9b2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:49:23 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:35014
I20260928 02:49:23.758584 13408 tablet_service.cc:1514] Processing DeleteTablet for tablet f77bcf35a1f7408eb032978ba9c9b2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:49:23 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:43380
I20260928 02:49:23.759049 13638 tablet_replica.cc:330] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: stopping tablet replica
I20260928 02:49:23.759188 13637 tablet_replica.cc:330] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: stopping tablet replica
I20260928 02:49:23.759241 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 13185
I20260928 02:49:23.759302 13637 raft_consensus.cc:2229] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:49:23.759950 13636 tablet_replica.cc:330] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: stopping tablet replica
I20260928 02:49:23.760210 13636 raft_consensus.cc:2229] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:49:23.759363 13638 raft_consensus.cc:2229] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:49:23.760576 13638 raft_consensus.cc:2258] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:49:23.760576 13636 raft_consensus.cc:2258] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:49:23.762074 13638 ts_tablet_manager.cc:1905] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:49:23.762089 13636 ts_tablet_manager.cc:1905] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:49:23.764655 13638 ts_tablet_manager.cc:1918] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:49:23.764657 13636 ts_tablet_manager.cc:1918] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:49:23.764734 13638 log.cc:1196] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:23.764752 13636 log.cc:1196] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:23.765027 13638 ts_tablet_manager.cc:1939] T f77bcf35a1f7408eb032978ba9c9b2dc P 48e9816a480742f2a03b31d175060094: Deleting consensus metadata
I20260928 02:49:23.765084 13636 ts_tablet_manager.cc:1939] T f77bcf35a1f7408eb032978ba9c9b2dc P 54b957f939f842edb7efc58b15598287: Deleting consensus metadata
I20260928 02:49:23.766139 13131 catalog_manager.cc:4841] TS 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083): tablet f77bcf35a1f7408eb032978ba9c9b2dc (table test-table [id=605b994dd62b442f889e3c0677388158]) successfully deleted
I20260928 02:49:23.766129 13132 catalog_manager.cc:4841] TS 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): tablet f77bcf35a1f7408eb032978ba9c9b2dc (table test-table [id=605b994dd62b442f889e3c0677388158]) successfully deleted
W20260928 02:49:23.768167 13131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:37797 (error 108)
I20260928 02:49:23.768702 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 13340
I20260928 02:49:23.771001 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:23.770666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:37797 (local address 127.0.0.1:35030)
0928 02:49:23.770818 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:23.770900 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:49:23.771183 13131 catalog_manager.cc:4584] TS 4aec09e3e9554ab991cac5aa8b525b72 (127.28.248.129:37797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f77bcf35a1f7408eb032978ba9c9b2dc: Network error: Client connection negotiation failed: client connection to 127.28.248.129:37797: connect: Connection refused (error 111)
I20260928 02:49:23.775002 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 13476
I20260928 02:49:23.779708 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 13114
I20260928 02:49:43.785528 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36499
--webserver_interface=127.28.248.190
--webserver_port=36397
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:36499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:49:43.897684 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:43.898003 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:43.898044 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:43.902067 13640 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:49:43.902118 13640 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:49:43.902133 13640 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:49:43.902148 13640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:49:43.902166 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:43.902225 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:43.902254 13640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:49:43.902284 13640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:49:43.907660 13640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:36499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=36397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:43.908838 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:43.909901 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:43.918902 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:43.918990 13640 server_base.cc:1034] running on GCE node
W20260928 02:49:43.918900 13648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:43.918974 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:43.919744 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:43.920977 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:43.922196 13640 hybrid_clock.cc:648] HybridClock initialized: now 1790563783922170 us; error 32 us; skew 500 ppm
Sep 28 02:49:43 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563783, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:43.925209 13640 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:49:43.926103 13640 webserver.cc:460] Webserver started at http://127.28.248.190:36397/ using document root <none> and password file <none>
I20260928 02:49:43.926707 13640 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:43.926801 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:43.930727 13640 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:49:43.933140 13655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:43.934626 13640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:49:43.934794 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "3924dd43804d4dfd829f284297958fbd"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "67962d569c9fb8e8dc525df161e14b76"
server_key_iv: "6e3436106fcaed1fdcb65918dac16650"
server_key_version: "encryptionkey@0"
I20260928 02:49:43.935395 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:43.958132 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:43.961687 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:43.961915 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:43.969409 13640 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:36499
I20260928 02:49:43.969709 13707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:36499 every 8 connection(s)
I20260928 02:49:43.970549 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:49:43.972692 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13640
I20260928 02:49:43.973163 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:37797
--local_ip_for_outbound_sockets=127.28.248.129
--tserver_master_addrs=127.28.248.190:36499
--webserver_port=36675
--webserver_interface=127.28.248.129
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:49:43.975508 13708 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:49:43.978298 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap starting.
I20260928 02:49:44.002750 13708 log.cc:824] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:44.017288 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:49:44.017797 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Bootstrap complete.
I20260928 02:49:44.023550 13708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:44.024133 13708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3924dd43804d4dfd829f284297958fbd, State: Initialized, Role: FOLLOWER
I20260928 02:49:44.024762 13708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:44.024922 13708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:49:44.024986 13708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:49:44.025086 13708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:49:44.027704 13708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:44.028088 13708 leader_election.cc:302] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3924dd43804d4dfd829f284297958fbd; no voters:
I20260928 02:49:44.028460 13708 leader_election.cc:288] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:49:44.028795 13712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:49:44.029048 13712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [term 3 LEADER]: Becoming Leader. State: Replica: 3924dd43804d4dfd829f284297958fbd, State: Running, Role: LEADER
I20260928 02:49:44.029371 13712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } }
I20260928 02:49:44.029838 13708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:49:44.030926 13714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3924dd43804d4dfd829f284297958fbd. Latest consensus state: current_term: 3 leader_uuid: "3924dd43804d4dfd829f284297958fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } } }
I20260928 02:49:44.031039 13714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:49:44.030897 13713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3924dd43804d4dfd829f284297958fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3924dd43804d4dfd829f284297958fbd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36499 } } }
I20260928 02:49:44.031164 13713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:49:44.031459 13721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:49:44.036761 13721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:49:44.037580 13721 catalog_manager.cc:1259] Loaded cluster ID: 904246e10a5f47ccbfaecd43e2dcf11b
I20260928 02:49:44.037655 13721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:49:44.040360 13721 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:49:44.041193 13721 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Loaded TSK: 0
I20260928 02:49:44.041281 13721 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3924dd43804d4dfd829f284297958fbd: Loaded TSK: 1
F20260928 02:49:44.042409 13721 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f75d75e39ad google::LogMessage::Fail() at ??:0
@ 0x7f75d75e688a google::LogMessage::SendToLog() at ??:0
@ 0x7f75d75e34de google::LogMessage::Flush() at ??:0
@ 0x7f75d75e4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f75da2a718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f75da29ff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f75e1f487bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f75e1f2299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f75e1f23178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f75e1f64df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f75e1f8382b std::function<>::operator()() at ??:0
@ 0x7f75e1f22b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f75e1f23f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f75e1f20579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f75e1f64640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f75d98353d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f75d9835c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f75d98373f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f75d9826512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f75dae6a6db start_thread at ??:0
@ 0x7f75d60e161f clone at ??:0
W20260928 02:49:44.127871 13710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:44.128120 13710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:44.128165 13710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:44.132784 13710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:44.132834 13710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:44.132897 13710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:49:44.138700 13710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:37797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=36675
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:44.139959 13710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:44.141034 13710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:44.151248 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:44.151388 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:44.151248 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:44.151538 13710 server_base.cc:1034] running on GCE node
I20260928 02:49:44.152060 13710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:44.152956 13710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:44.154158 13710 hybrid_clock.cc:648] HybridClock initialized: now 1790563784154130 us; error 46 us; skew 500 ppm
Sep 28 02:49:44 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563784, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:44.157729 13710 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:49:44.158730 13710 webserver.cc:460] Webserver started at http://127.28.248.129:36675/ using document root <none> and password file <none>
I20260928 02:49:44.159348 13710 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:44.159447 13710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:44.164136 13710 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:49:44.166860 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.168356 13710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:49:44.168525 13710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "4aec09e3e9554ab991cac5aa8b525b72"
format_stamp: "Formatted at 2026-09-28 02:49:01 on dist-test-slave-r238"
server_key: "587d29a648481f4422c3750e1d0b9aa0"
server_key_iv: "a931cc839140180252992b789faddbd4"
server_key_version: "encryptionkey@0"
I20260928 02:49:44.169234 13710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:44.182297 13710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:44.185806 13710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:44.186029 13710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:44.186736 13710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:44.187915 13772 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:49:44.193810 13710 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:49:44.193923 13710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 02:49:44.194003 13710 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:49:44.195751 13710 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:49:44.195827 13710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 02:49:44.196132 13772 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap starting.
I20260928 02:49:44.206491 13710 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:37797
I20260928 02:49:44.206753 13879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:37797 every 8 connection(s)
I20260928 02:49:44.207549 13710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:49:44.212889 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13710
I20260928 02:49:44.213346 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:39083
--local_ip_for_outbound_sockets=127.28.248.130
--tserver_master_addrs=127.28.248.190:36499
--webserver_port=37847
--webserver_interface=127.28.248.130
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:49:44.220005 13772 log.cc:824] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Log is configured to *not* fsync() on all Append() calls
I20260928 02:49:44.227367 13772 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:49:44.227838 13772 tablet_bootstrap.cc:492] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Bootstrap complete.
I20260928 02:49:44.229063 13772 ts_tablet_manager.cc:1397] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.011s
I20260928 02:49:44.231528 13772 raft_consensus.cc:348] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:44.231974 13772 raft_consensus.cc:729] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aec09e3e9554ab991cac5aa8b525b72, State: Initialized, Role: FOLLOWER
I20260928 02:49:44.232474 13772 consensus_queue.cc:260] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:44.233482 13772 ts_tablet_manager.cc:1428] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260928 02:49:44.314360 13882 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:49:44.209600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:42905)
0928 02:49:44.210093 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.210149 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:44.210995 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:44.313438 (+102443us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
W20260928 02:49:44.315026 13880 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:36499 (0 consecutive failures): Network error: Failed to ping master at 127.28.248.190:36499: Client connection negotiation failed: client connection to 127.28.248.190:36499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:49:44.315807 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.315530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:39021)
0928 02:49:44.315664 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.315737 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:49:44.316679 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.316365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:56343)
0928 02:49:44.316505 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.316555 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 02:49:44.316860 13880 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:49:44.353081 13883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:44.353398 13883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:44.353478 13883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:44.357712 13883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:44.357808 13883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:44.357908 13883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:49:44.363301 13883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:39083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=37847
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:44.364426 13883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:44.365502 13883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:44.375459 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:44.375478 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:44.375667 13883 server_base.cc:1034] running on GCE node
W20260928 02:49:44.375471 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:44.376166 13883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:44.376879 13883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:44.378083 13883 hybrid_clock.cc:648] HybridClock initialized: now 1790563784378059 us; error 35 us; skew 500 ppm
Sep 28 02:49:44 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563784, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:44.380916 13883 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:49:44.381819 13883 webserver.cc:460] Webserver started at http://127.28.248.130:37847/ using document root <none> and password file <none>
I20260928 02:49:44.382336 13883 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:44.382395 13883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:44.386260 13883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:49:44.388460 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.389849 13883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:49:44.389952 13883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "48e9816a480742f2a03b31d175060094"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "1b30d7835059a8dca36c0fbeb8f719e8"
server_key_iv: "49e0e534d5fcdf598b669d974385c999"
server_key_version: "encryptionkey@0"
I20260928 02:49:44.390487 13883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:44.444142 13883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:44.447216 13883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:44.447378 13883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:44.447923 13883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:44.448733 13883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:49:44.448779 13883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.448819 13883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:44.448835 13883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.458091 13883 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:39083
I20260928 02:49:44.458115 14013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:39083 every 8 connection(s)
I20260928 02:49:44.459317 13883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:49:44.462527 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 13883
I20260928 02:49:44.462956 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:43933
--local_ip_for_outbound_sockets=127.28.248.131
--tserver_master_addrs=127.28.248.190:36499
--webserver_port=40255
--webserver_interface=127.28.248.131
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:49:44.463102 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.461540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:44807)
0928 02:49:44.462195 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.462303 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":110,"threads_started":1}
W20260928 02:49:44.463523 14014 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:36499 (0 consecutive failures): Network error: Failed to ping master at 127.28.248.190:36499: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
I20260928 02:49:44.464131 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.463928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:43949)
0928 02:49:44.464036 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.464082 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:49:44.464823 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.464548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:37777)
0928 02:49:44.464699 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.464756 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 02:49:44.465023 14014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:49:44.574944 14017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:49:44.575209 14017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:49:44.575254 14017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:49:44.579381 14017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:49:44.579444 14017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:49:44.579504 14017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:49:44.584904 14017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:43933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=40255
--tserver_master_addrs=127.28.248.190:36499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:49:44.586000 14017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:49:44.587039 14017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:49:44.597174 14022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:49:44.597289 14025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:44.597434 14017 server_base.cc:1034] running on GCE node
W20260928 02:49:44.597289 14023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:49:44.597888 14017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:49:44.598536 14017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:49:44.599712 14017 hybrid_clock.cc:648] HybridClock initialized: now 1790563784599665 us; error 62 us; skew 500 ppm
Sep 28 02:49:44 dist-test-slave-r238 krb5kdc[12567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563784, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:49:44.602595 14017 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:49:44.603462 14017 webserver.cc:460] Webserver started at http://127.28.248.131:40255/ using document root <none> and password file <none>
I20260928 02:49:44.604022 14017 fs_manager.cc:359] Metadata directory not provided
I20260928 02:49:44.604084 14017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:49:44.607988 14017 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:49:44.610235 14032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.611550 14017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:49:44.611699 14017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "54b957f939f842edb7efc58b15598287"
format_stamp: "Formatted at 2026-09-28 02:49:02 on dist-test-slave-r238"
server_key: "7cf2875b94cd6767f98a3d27a4b46c18"
server_key_iv: "240d757252c80f36b18a4bcf91d3d0fa"
server_key_version: "encryptionkey@0"
I20260928 02:49:44.612259 14017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:49:44.627067 14017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:49:44.630594 14017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:49:44.630815 14017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:49:44.631458 14017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:49:44.632299 14017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:49:44.632372 14017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.632444 14017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:49:44.632496 14017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:49:44.641858 14017 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:43933
I20260928 02:49:44.641881 14145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:43933 every 8 connection(s)
I20260928 02:49:44.642901 14017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:49:44.647122 14148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.645272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:55125)
0928 02:49:44.645893 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.646366 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":130,"threads_started":1}
W20260928 02:49:44.647835 14146 heartbeater.cc:644] Failed to heartbeat to 127.28.248.190:36499 (0 consecutive failures): Network error: Failed to ping master at 127.28.248.190:36499: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
I20260928 02:49:44.648524 14148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.648286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:55963)
0928 02:49:44.648412 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.648465 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:49:44.649261 14148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:44.649078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:41921)
0928 02:49:44.649172 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:44.649214 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 02:49:44.649435 14146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:49:44.649967 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14017
I20260928 02:49:45.318009 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.317476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:56915)
0928 02:49:45.317836 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:45.317894 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
I20260928 02:49:45.466073 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.465646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:34153)
0928 02:49:45.465904 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:45.465965 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260928 02:49:45.650449 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.649984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:59729)
0928 02:49:45.650267 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:45.650327 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260928 02:49:45.816365 14153 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:45.816776 14153 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:45.818101 14153 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
Sep 28 02:49:45 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563784, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.131@KRBTEST.COM
Sep 28 02:49:45 dist-test-slave-r238 krb5kdc[12567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563784, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.130@KRBTEST.COM
I20260928 02:49:45.832949 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.818407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:43933 (local address 127.28.248.129:38753)
0928 02:49:45.818758 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:45.818772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:45.818841 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:45.820540 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:45.820547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:45.821129 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:45.821621 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:45.821638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.822975 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.822979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:45.824034 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:45.824046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.824880 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.824883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:45.825105 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:45.826053 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:45.826071 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:45.827880 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:45.830295 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:45.830303 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:45.830318 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:45.830593 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:45.831017 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:45.831022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:45.831024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:45.831173 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:45.831878 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:45.831885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:45.832136 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 02:49:45.832338 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 02:49:45.832761 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":192,"spinlock_wait_cycles":6528,"thread_start_us":68,"threads_started":1}
I20260928 02:49:45.832966 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.818709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:38753 (local address 127.28.248.131:43933)
0928 02:49:45.819072 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:45.819091 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:45.819123 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:49:45.819759 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:45.819764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:45.820128 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:45.820468 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:45.821817 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.822808 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.824171 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.824746 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.824924 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:45.828173 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:45.828178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:45.828194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:45.828227 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:45.830150 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:45.830776 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:45.830781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:45.830785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:45.830910 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:45.831276 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:45.831281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:45.831283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:45.831754 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:45.831905 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:45.832451 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 02:49:45.832754 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260928 02:49:45.832986 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.818510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:42381 (local address 127.28.248.130:39083)
0928 02:49:45.818989 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:49:45.819022 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 02:49:45.819061 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:49:45.819760 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:49:45.819764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:49:45.820128 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:49:45.820439 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:49:45.821818 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.822808 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.824180 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.824745 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.824924 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:45.828435 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:49:45.828440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:49:45.828452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:49:45.828516 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:49:45.830462 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:45.830921 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:45.830926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:45.830932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:45.831058 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:49:45.831367 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:49:45.831372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:49:45.831374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:49:45.831763 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:49:45.831905 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:49:45.832452 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 02:49:45.832786 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":74,"threads_started":1}
I20260928 02:49:45.833024 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:45.818352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:39083 (local address 127.28.248.129:42381)
0928 02:49:45.818696 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:45.818712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:49:45.818828 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:49:45.820540 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:49:45.820547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:49:45.821129 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:49:45.821622 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:45.821639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.822967 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.822975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:45.824039 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:49:45.824050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:49:45.824880 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:49:45.824883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:49:45.825107 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:49:45.826719 (+ 1612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:49:45.826740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:49:45.828216 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:49:45.830590 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:45.830595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:45.830597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:45.830803 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:45.831157 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:49:45.831161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:49:45.831163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:49:45.831255 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:49:45.831879 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:49:45.831886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:49:45.832136 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 02:49:45.832339 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 02:49:45.832849 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":368,"spinlock_wait_cycles":34944,"thread_start_us":84,"threads_started":1}
W20260928 02:49:45.835855 13766 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
W20260928 02:49:45.836083 13767 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:45.836162 13767 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72; no voters: 48e9816a480742f2a03b31d175060094, 54b957f939f842edb7efc58b15598287
I20260928 02:49:45.836470 14153 raft_consensus.cc:2726] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:49:46.319175 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:46.318852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:55389)
0928 02:49:46.319013 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:46.319087 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:49:46.467527 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:46.467031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:50247)
0928 02:49:46.467365 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:46.467431 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260928 02:49:46.651774 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:46.651322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:58047)
0928 02:49:46.651577 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:46.651670 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260928 02:49:47.320441 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:47.320009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:40737)
0928 02:49:47.320282 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:47.320343 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260928 02:49:47.469054 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:47.468483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:52163)
0928 02:49:47.468876 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:47.468940 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":171,"threads_started":1}
I20260928 02:49:47.653198 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:47.652739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:58123)
0928 02:49:47.653007 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:47.653077 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260928 02:49:48.079788 14163 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:48.079952 14163 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:48.080490 14163 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
W20260928 02:49:48.081421 13766 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
W20260928 02:49:48.081537 13767 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:48.081594 13767 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72; no voters: 48e9816a480742f2a03b31d175060094, 54b957f939f842edb7efc58b15598287
I20260928 02:49:48.081780 14163 raft_consensus.cc:2726] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:49:48.321794 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:48.321340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:33807)
0928 02:49:48.321629 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:48.321695 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260928 02:49:48.470398 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:48.469878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:44259)
0928 02:49:48.470212 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:48.470277 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":91,"threads_started":1}
I20260928 02:49:48.654502 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:48.654105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:50863)
0928 02:49:48.654348 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:48.654410 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260928 02:49:49.323271 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:49.322790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:49035)
0928 02:49:49.323097 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:49.323174 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
I20260928 02:49:49.471765 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:49.471299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:55971)
0928 02:49:49.471597 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:49.471667 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260928 02:49:49.655817 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:49.655360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:33813)
0928 02:49:49.655623 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:49.655679 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260928 02:49:50.292124 14170 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:50.292320 14170 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:50.292870 14170 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
W20260928 02:49:50.293889 13766 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
W20260928 02:49:50.293987 13767 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:50.294097 13767 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72; no voters: 48e9816a480742f2a03b31d175060094, 54b957f939f842edb7efc58b15598287
I20260928 02:49:50.294253 14170 raft_consensus.cc:2726] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:49:50.324810 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:50.324183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:38941)
0928 02:49:50.324484 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:50.324692 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20260928 02:49:50.473237 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:50.472768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:36449)
0928 02:49:50.473057 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:50.473123 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260928 02:49:50.657279 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:50.656867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:42619)
0928 02:49:50.657116 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:50.657180 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260928 02:49:51.326115 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:51.325651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:46393)
0928 02:49:51.325953 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:51.326017 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260928 02:49:51.474694 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:51.474150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:42911)
0928 02:49:51.474527 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:51.474585 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":115,"threads_started":1}
I20260928 02:49:51.658597 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:51.658113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:35703)
0928 02:49:51.658440 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:51.658503 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260928 02:49:52.327518 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:52.327052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:59579)
0928 02:49:52.327343 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:52.327414 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260928 02:49:52.475988 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:52.475546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:33229)
0928 02:49:52.475825 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:52.475890 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260928 02:49:52.659999 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:52.659541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:44403)
0928 02:49:52.659821 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:52.659888 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260928 02:49:53.328847 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:53.328354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:50875)
0928 02:49:53.328682 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:53.328747 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260928 02:49:53.477308 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:53.476842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:36391)
0928 02:49:53.477110 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:53.477173 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260928 02:49:53.661466 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:53.660945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:49007)
0928 02:49:53.661296 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:53.661367 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260928 02:49:53.996556 14183 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:53.996821 14183 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:53.997319 14183 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
W20260928 02:49:53.998057 13766 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
W20260928 02:49:53.998203 13767 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:53.998314 13767 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72; no voters: 48e9816a480742f2a03b31d175060094, 54b957f939f842edb7efc58b15598287
I20260928 02:49:53.998478 14183 raft_consensus.cc:2726] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:49:54.330077 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:54.329685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:36737)
0928 02:49:54.329928 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:54.329981 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260928 02:49:54.478751 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:54.478261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:45479)
0928 02:49:54.478541 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:54.478610 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260928 02:49:54.662689 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:54.662249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:45297)
0928 02:49:54.662524 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:54.662583 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
I20260928 02:49:55.331391 14187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:55.330899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:46739)
0928 02:49:55.331203 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:55.331279 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260928 02:49:55.480180 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:55.479711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:36189)
0928 02:49:55.480008 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:55.480075 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20260928 02:49:55.664172 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:55.663677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:46669)
0928 02:49:55.663970 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:55.664041 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260928 02:49:56.332998 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:56.332417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:46911)
0928 02:49:56.332829 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:56.332899 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":109,"threads_started":1}
I20260928 02:49:56.481806 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:56.481309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:56027)
0928 02:49:56.481630 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:56.481704 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260928 02:49:56.665796 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:56.665144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:32897)
0928 02:49:56.665572 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:56.665668 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":146,"threads_started":1}
I20260928 02:49:57.334370 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:57.333898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:46435)
0928 02:49:57.334208 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:57.334271 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
I20260928 02:49:57.483327 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:57.482849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:46523)
0928 02:49:57.483155 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:57.483225 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20260928 02:49:57.667060 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:57.666650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:60461)
0928 02:49:57.666899 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:57.666959 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260928 02:49:58.044760 14196 raft_consensus.cc:482] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:49:58.044955 14196 raft_consensus.cc:504] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e9816a480742f2a03b31d175060094" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 39083 } } peers { permanent_uuid: "54b957f939f842edb7efc58b15598287" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 43933 } } peers { permanent_uuid: "4aec09e3e9554ab991cac5aa8b525b72" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 37797 } }
I20260928 02:49:58.045495 14196 leader_election.cc:288] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083), 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933)
W20260928 02:49:58.046204 13766 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48e9816a480742f2a03b31d175060094 (127.28.248.130:39083): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
W20260928 02:49:58.046293 13767 leader_election.cc:341] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 54b957f939f842edb7efc58b15598287 (127.28.248.131:43933): Not found: Tablet not found: f77bcf35a1f7408eb032978ba9c9b2dc
I20260928 02:49:58.046345 13766 leader_election.cc:302] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4aec09e3e9554ab991cac5aa8b525b72; no voters: 48e9816a480742f2a03b31d175060094, 54b957f939f842edb7efc58b15598287
I20260928 02:49:58.046551 14196 raft_consensus.cc:2726] T f77bcf35a1f7408eb032978ba9c9b2dc P 4aec09e3e9554ab991cac5aa8b525b72 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:49:58.335722 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:58.335280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:55897)
0928 02:49:58.335554 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:58.335620 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
I20260928 02:49:58.484829 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:58.484247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:48071)
0928 02:49:58.484577 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:58.484670 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
I20260928 02:49:58.668368 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:58.667941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.131:49657)
0928 02:49:58.668202 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:58.668274 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260928 02:49:59.337342 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:59.336917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.129:43429)
0928 02:49:59.337181 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:59.337244 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260928 02:49:59.486269 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:49:59.485820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36499 (local address 127.28.248.130:57857)
0928 02:49:59.486105 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:49:59.486173 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.190:36499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260928 02:49:59.652328 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 13710
I20260928 02:49:59.658141 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 13883
I20260928 02:49:59.664186 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14017
2026-09-28T02:49:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59260 ms)
[----------] 30 tests from SecurityITest (239118 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:49:59 dist-test-slave-r238 krb5kdc[14204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:49:59 dist-test-slave-r238 krb5kdc[14204](info): set up 2 sockets
Sep 28 02:49:59 dist-test-slave-r238 krb5kdc[14204](info): commencing operation
krb5kdc: starting...
W20260928 02:50:00.721416 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:00 dist-test-slave-r238 krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:00.853739 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46777
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:00.968003 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:00.968320 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:00.968364 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:00.972525 14217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:00.972620 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:00.972643 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:00.972661 14217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:00.972738 14217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:00.978367 14217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:46777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:00.979601 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:00.980760 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:00.990206 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:00.990221 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:00.990254 14217 server_base.cc:1034] running on GCE node
W20260928 02:50:00.990258 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:00.991243 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:00.992777 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:00.993979 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790563800993945 us; error 52 us; skew 500 ppm
Sep 28 02:50:00 dist-test-slave-r238 krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563800, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:00.996874 14217 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:00.997931 14217 webserver.cc:460] Webserver started at http://127.28.248.190:45671/ using document root <none> and password file <none>
I20260928 02:50:00.998548 14217 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:00.998646 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:00.998888 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:01.001091 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "26937672808d4d39898d513f40e710cd"
format_stamp: "Formatted at 2026-09-28 02:50:00 on dist-test-slave-r238"
server_key: "eb1e105e713542c67a6476b45e0690ad"
server_key_iv: "936988fe733e2438ad4c933ec6a56983"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.001694 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "26937672808d4d39898d513f40e710cd"
format_stamp: "Formatted at 2026-09-28 02:50:00 on dist-test-slave-r238"
server_key: "eb1e105e713542c67a6476b45e0690ad"
server_key_iv: "936988fe733e2438ad4c933ec6a56983"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.005863 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:50:01.008571 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.010085 14217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:50:01.010319 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "26937672808d4d39898d513f40e710cd"
format_stamp: "Formatted at 2026-09-28 02:50:00 on dist-test-slave-r238"
server_key: "eb1e105e713542c67a6476b45e0690ad"
server_key_iv: "936988fe733e2438ad4c933ec6a56983"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.010474 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:01.023129 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:01.027595 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:01.027871 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:01.038053 14217 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:46777
I20260928 02:50:01.038318 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:46777 every 8 connection(s)
I20260928 02:50:01.039237 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:01.040999 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14217
I20260928 02:50:01.041126 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:01.041445 29666 external_mini_cluster.cc:1428] Setting key c1343a745b1f68ec504e5c9e742cba87
I20260928 02:50:01.042677 14285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:01.049317 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd: Bootstrap starting.
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:01.052008 14285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:01.052959 14285 log.cc:824] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:01.054946 14285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd: No bootstrap required, opened a new log
I20260928 02:50:01.056186 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.042957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.28.248.190:46777)
0928 02:50:01.043382 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.043397 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.043431 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:50:01.044083 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.044099 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.044439 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.044801 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:01.045670 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.046668 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.047845 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.048337 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.048459 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.051300 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:01.051305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:01.051319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:01.051348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:01.053264 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.053788 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.053801 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.053806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.053918 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.054220 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.054224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.054225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.054633 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:01.054837 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.055126 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.055441 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":136,"threads_started":1}
I20260928 02:50:01.059221 14285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26937672808d4d39898d513f40e710cd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46777 } }
I20260928 02:50:01.059479 14285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:01.059567 14285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26937672808d4d39898d513f40e710cd, State: Initialized, Role: FOLLOWER
I20260928 02:50:01.060021 14285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26937672808d4d39898d513f40e710cd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46777 } }
I20260928 02:50:01.060179 14285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:01.060257 14285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:01.060385 14285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:01.061628 14285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26937672808d4d39898d513f40e710cd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46777 } }
I20260928 02:50:01.062016 14285 leader_election.cc:302] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26937672808d4d39898d513f40e710cd; no voters:
I20260928 02:50:01.062356 14285 leader_election.cc:288] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:01.062495 14290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:01.062765 14290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [term 1 LEADER]: Becoming Leader. State: Replica: 26937672808d4d39898d513f40e710cd, State: Running, Role: LEADER
I20260928 02:50:01.063135 14290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26937672808d4d39898d513f40e710cd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46777 } }
I20260928 02:50:01.063628 14285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:01.064946 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26937672808d4d39898d513f40e710cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26937672808d4d39898d513f40e710cd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46777 } } }
I20260928 02:50:01.064991 14292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26937672808d4d39898d513f40e710cd. Latest consensus state: current_term: 1 leader_uuid: "26937672808d4d39898d513f40e710cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26937672808d4d39898d513f40e710cd" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46777 } } }
I20260928 02:50:01.065068 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:01.065101 14292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:01.065469 14299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:01.068403 14299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:01.074369 14299 catalog_manager.cc:1347] Generated new cluster ID: ea794a3b6488409db6a59924b33d74e9
I20260928 02:50:01.074465 14299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:01.090695 14299 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:01.092046 14299 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:01.101960 14299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd: Generated new TSK 0
I20260928 02:50:01.102883 14299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:50:01.139300 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46777
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:01.256173 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:01.256464 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:01.256510 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:01.260720 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:01.260797 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:01.260865 14311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:50:01.266453 14311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:01.267627 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:01.268927 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:01.279636 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:01.279634 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:01.279701 14311 server_base.cc:1034] running on GCE node
W20260928 02:50:01.280098 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:01.280594 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:01.281350 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:01.282518 14311 hybrid_clock.cc:648] HybridClock initialized: now 1790563801282502 us; error 32 us; skew 500 ppm
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:01.285435 14311 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:50:01.286417 14311 webserver.cc:460] Webserver started at http://127.28.248.129:36031/ using document root <none> and password file <none>
I20260928 02:50:01.287032 14311 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:01.287129 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:01.287374 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:01.289588 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "6e44316354b54979b796a123a494880f"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "adf9cc6acf29865eb0d10819dd27c08e"
server_key_iv: "7a00addbb5c446903f44a3821ec76b2d"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.290177 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "6e44316354b54979b796a123a494880f"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "adf9cc6acf29865eb0d10819dd27c08e"
server_key_iv: "7a00addbb5c446903f44a3821ec76b2d"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.294082 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:01.296542 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.297892 14311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:01.298041 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "6e44316354b54979b796a123a494880f"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "adf9cc6acf29865eb0d10819dd27c08e"
server_key_iv: "7a00addbb5c446903f44a3821ec76b2d"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.298163 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:01.320394 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:01.323679 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:01.323911 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:01.324534 14311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:01.325428 14311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:01.325502 14311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.325574 14311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:01.325625 14311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.334949 14311 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:34545
I20260928 02:50:01.334990 14439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:34545 every 8 connection(s)
I20260928 02:50:01.335978 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:01.336838 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14311
I20260928 02:50:01.336993 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:01.337321 29666 external_mini_cluster.cc:1428] Setting key 87d3e640e503ac749afb2233f70deaa4
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:01.356177 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.338319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:44091 (local address 127.28.248.190:46777)
0928 02:50:01.338473 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.338478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.339547 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 02:50:01.341658 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.341664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.341733 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.341838 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:01.343169 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.344107 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.345433 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.345838 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.345961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.351675 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:01.351679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:01.351681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:01.351711 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:01.353571 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.354234 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.354241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.354246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.354358 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.354725 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.354732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.354736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.355027 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:01.355146 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.355755 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.355920 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:50:01.356951 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.338737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46777 (local address 127.28.248.129:44091)
0928 02:50:01.339354 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.339417 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.341401 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.342064 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.342072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.342472 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:01.342989 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.343004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.344268 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.344271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.345295 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.345306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.346024 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.346028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.346201 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.346899 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:01.346917 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:01.351430 (+ 4513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:01.353751 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:01.353757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:01.353773 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:01.354093 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:01.354458 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:01.354461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:01.354462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:01.354613 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:01.355178 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:01.355184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:01.355483 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.355648 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.355940 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":152,"threads_started":1}
I20260928 02:50:01.358335 14440 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46777
I20260928 02:50:01.358603 14440 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:01.359144 14440 heartbeater.cc:511] Master 127.28.248.190:46777 requested a full tablet report, sending...
I20260928 02:50:01.360900 14243 ts_manager.cc:194] Registered new tserver with Master: 6e44316354b54979b796a123a494880f (127.28.248.129:34545)
I20260928 02:50:01.362089 14243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:44091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:50:01.371869 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46777
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:01.488363 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:01.488739 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:01.488828 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:01.493122 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:01.493191 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:01.493279 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:50:01.498775 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:01.499955 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:01.501186 14445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:01.511374 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:01.511380 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:01.511430 14445 server_base.cc:1034] running on GCE node
W20260928 02:50:01.511713 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:01.512142 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:01.512885 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:01.514081 14445 hybrid_clock.cc:648] HybridClock initialized: now 1790563801514055 us; error 39 us; skew 500 ppm
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:01.517118 14445 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:50:01.518035 14445 webserver.cc:460] Webserver started at http://127.28.248.130:44389/ using document root <none> and password file <none>
I20260928 02:50:01.518661 14445 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:01.518754 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:01.518988 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:01.521245 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "bcc5db68cedc438f9edccae03ab279a4"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "584dd6a14285cb917af06b874f1b7f6e"
server_key_iv: "7c300164fefd0579b1a7fd4198fa5090"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.521828 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "bcc5db68cedc438f9edccae03ab279a4"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "584dd6a14285cb917af06b874f1b7f6e"
server_key_iv: "7c300164fefd0579b1a7fd4198fa5090"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.525684 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:01.528262 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.529619 14445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:01.529717 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "bcc5db68cedc438f9edccae03ab279a4"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "584dd6a14285cb917af06b874f1b7f6e"
server_key_iv: "7c300164fefd0579b1a7fd4198fa5090"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.529865 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:01.554899 14445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:01.558442 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:01.558684 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:01.559321 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:01.560194 14445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:01.560243 14445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.560283 14445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:01.560352 14445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.570000 14445 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:45001
I20260928 02:50:01.570019 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:45001 every 8 connection(s)
I20260928 02:50:01.571009 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:01.579470 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14445
I20260928 02:50:01.579612 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:01.579900 29666 external_mini_cluster.cc:1428] Setting key 7267fc8b68afe1bb50da41ad65315544
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:01.585685 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.573069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:45181 (local address 127.28.248.190:46777)
0928 02:50:01.573208 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.573212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.574065 (+ 853us) server_negotiation.cc:407] Connection header received
0928 02:50:01.575405 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.575409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.575464 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.575546 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:01.576820 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.577373 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.578499 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.578730 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.578813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.581839 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:01.581843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:01.581845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:01.581870 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:01.583432 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.584002 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.584006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.584007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.584091 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.584467 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.584472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.584474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.584746 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:01.584854 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.585393 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.585523 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:50:01.586617 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.573468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46777 (local address 127.28.248.130:45181)
0928 02:50:01.573912 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.573957 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.575176 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.575715 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.575722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.576088 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:01.576586 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.576599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.577489 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.577491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.578385 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.578395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.578870 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.578873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.579015 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.579777 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:01.579794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:01.581640 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:01.583574 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:01.583581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:01.583599 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:01.583887 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:01.584189 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:01.584192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:01.584194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:01.584365 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:01.584867 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:01.584876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:01.585146 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.585349 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.585740 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260928 02:50:01.587929 14574 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46777
I20260928 02:50:01.588207 14574 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:01.588791 14574 heartbeater.cc:511] Master 127.28.248.190:46777 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:50:01.589944 14243 ts_manager.cc:194] Registered new tserver with Master: bcc5db68cedc438f9edccae03ab279a4 (127.28.248.130:45001)
I20260928 02:50:01.590539 14243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:45181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:50:01.608409 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46777
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:01.722734 14579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:01.723021 14579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:01.723065 14579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:01.727140 14579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:01.727208 14579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:01.727273 14579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:50:01.732599 14579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:46777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:01.733760 14579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:01.734853 14579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:01.744872 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:01.744927 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:01.745056 14579 server_base.cc:1034] running on GCE node
W20260928 02:50:01.744943 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:01.745590 14579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:01.746285 14579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:01.747495 14579 hybrid_clock.cc:648] HybridClock initialized: now 1790563801747474 us; error 34 us; skew 500 ppm
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:01.750504 14579 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:50:01.751375 14579 webserver.cc:460] Webserver started at http://127.28.248.131:46105/ using document root <none> and password file <none>
I20260928 02:50:01.751926 14579 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:01.751986 14579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:01.752149 14579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:01.754303 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "6a2e649e924a489981c922d9d0208b7a"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "a4a538cb4fda528deae55ab19a8122fa"
server_key_iv: "1c66a3b00649b893819a5db57d7a59b7"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.754848 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "6a2e649e924a489981c922d9d0208b7a"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "a4a538cb4fda528deae55ab19a8122fa"
server_key_iv: "1c66a3b00649b893819a5db57d7a59b7"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.758767 14579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:50:01.761276 14594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.762530 14579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:01.762631 14579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "6a2e649e924a489981c922d9d0208b7a"
format_stamp: "Formatted at 2026-09-28 02:50:01 on dist-test-slave-r238"
server_key: "a4a538cb4fda528deae55ab19a8122fa"
server_key_iv: "1c66a3b00649b893819a5db57d7a59b7"
server_key_version: "encryptionkey@0"
I20260928 02:50:01.762722 14579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:01.791474 14579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:01.794646 14579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:01.794842 14579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:01.795493 14579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:01.796343 14579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:01.796392 14579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.796434 14579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:01.796459 14579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:01.806133 14579 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:34453
I20260928 02:50:01.806156 14707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:34453 every 8 connection(s)
I20260928 02:50:01.807104 14579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:01.815514 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14579
I20260928 02:50:01.815678 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:01.815963 29666 external_mini_cluster.cc:1428] Setting key 8e8f12e165f078a7c0cf709bb0ab08d0
Sep 28 02:50:01 dist-test-slave-r238 krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563801, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:01.821246 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.809020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:46281 (local address 127.28.248.190:46777)
0928 02:50:01.809175 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.809180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.810104 (+ 924us) server_negotiation.cc:407] Connection header received
0928 02:50:01.811381 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.811383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.811431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.811508 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:01.812705 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.813245 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.814402 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.814674 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.814756 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.817680 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:01.817684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:01.817687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:01.817716 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:01.819077 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.819616 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.819619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.819621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.819702 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.820048 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.820053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.820054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.820261 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:01.820382 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.820953 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.821083 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:50:01.822183 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.809489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:46777 (local address 127.28.248.131:46281)
0928 02:50:01.809948 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.809991 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.811157 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.811673 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.811680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.812028 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:01.812527 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.812539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.813371 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.813374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.814290 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.814299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.814806 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.814810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.814968 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.815653 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:01.815680 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:01.817512 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:01.819197 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:01.819204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:01.819221 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:01.819510 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:01.819800 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:01.819804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:01.819806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:01.819953 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:01.820379 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:01.820387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:01.820710 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.820914 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.821256 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":122,"threads_started":1}
I20260928 02:50:01.823500 14708 heartbeater.cc:347] Connected to a master server at 127.28.248.190:46777
I20260928 02:50:01.823798 14708 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:01.824347 14708 heartbeater.cc:511] Master 127.28.248.190:46777 requested a full tablet report, sending...
I20260928 02:50:01.825634 14243 ts_manager.cc:194] Registered new tserver with Master: 6a2e649e924a489981c922d9d0208b7a (127.28.248.131:34453)
I20260928 02:50:01.826292 14243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:46281
I20260928 02:50:01.830328 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:01.840415 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.832475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.28.248.190:46777)
0928 02:50:01.832660 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.832665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.832871 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:50:01.833061 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.833070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.833113 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.833198 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:01.833936 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.834512 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.835351 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.835569 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.835680 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.837303 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:01.837307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:01.837309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:01.837332 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:01.838625 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.839062 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.839067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.839069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.839150 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:01.839471 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:01.839475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:01.839477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:01.839715 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:01.839867 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.840112 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.840245 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:01.843777 14243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:01.847033 14243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:01.861773 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.856177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.28.248.129:34545)
0928 02:50:01.856484 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.856498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.856520 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:50:01.856598 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.856628 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.856833 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.857283 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:01.857467 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.858344 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.860354 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.861011 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.861153 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.861298 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.861497 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.861601 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260928 02:50:01.861794 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.855716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34545 (local address 127.0.0.1:56870)
0928 02:50:01.856154 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.856167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.856250 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.856991 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.856996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.857016 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:01.857144 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.857154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.858457 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.858462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.860195 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.860206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.861136 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.861139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.861266 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.861318 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.861463 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.861588 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":236,"thread_start_us":123,"threads_started":1}
I20260928 02:50:01.862010 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.855326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:34453 (local address 127.0.0.1:52448)
0928 02:50:01.855706 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.855766 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.855902 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.856713 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.856722 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.856782 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:01.856946 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.856955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.858602 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.858605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.860195 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.860206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.861242 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.861246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.861344 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.861366 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.861421 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.861570 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":106,"thread_start_us":108,"threads_started":1}
I20260928 02:50:01.862104 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.856319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.28.248.130:45001)
0928 02:50:01.856960 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.856971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.857284 (+ 313us) server_negotiation.cc:407] Connection header received
0928 02:50:01.857359 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.857364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.857528 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.857673 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:01.857904 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.858881 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.860218 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.861102 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.861236 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.861379 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.861874 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.861973 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":116,"threads_started":1}
I20260928 02:50:01.862165 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.855525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.28.248.131:34453)
0928 02:50:01.855987 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.856154 (+ 167us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.856181 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:50:01.856267 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.856272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.856498 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.857118 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:01.857269 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.858469 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.860304 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.861107 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.861786 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.861907 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.861951 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.862032 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":136,"threads_started":1}
I20260928 02:50:01.861811 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.856077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:45001 (local address 127.0.0.1:49082)
0928 02:50:01.857093 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.857107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.857184 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.857674 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.857677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.857696 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:01.857774 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.857779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.859008 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.859014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.860095 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.860104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.861435 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.861438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.861518 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.861537 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.861602 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.861660 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":509,"thread_start_us":83,"threads_started":1}
I20260928 02:50:01.865199 14508 tablet_service.cc:1467] Processing CreateTablet for tablet 85aeb8f749b04623b33de3bfce75a695 (DEFAULT_TABLE table=test-table [id=fa35a42c78784383a43ce46ccfb9b4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:01.865264 14374 tablet_service.cc:1467] Processing CreateTablet for tablet 85aeb8f749b04623b33de3bfce75a695 (DEFAULT_TABLE table=test-table [id=fa35a42c78784383a43ce46ccfb9b4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:01.865264 14642 tablet_service.cc:1467] Processing CreateTablet for tablet 85aeb8f749b04623b33de3bfce75a695 (DEFAULT_TABLE table=test-table [id=fa35a42c78784383a43ce46ccfb9b4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:01.866371 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85aeb8f749b04623b33de3bfce75a695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:01.866369 14642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85aeb8f749b04623b33de3bfce75a695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:01.866369 14374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85aeb8f749b04623b33de3bfce75a695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:01.873044 14725 tablet_bootstrap.cc:492] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: Bootstrap starting.
I20260928 02:50:01.874389 14727 tablet_bootstrap.cc:492] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: Bootstrap starting.
I20260928 02:50:01.874636 14726 tablet_bootstrap.cc:492] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Bootstrap starting.
I20260928 02:50:01.875496 14725 tablet_bootstrap.cc:654] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:01.876803 14725 log.cc:824] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:01.876886 14727 tablet_bootstrap.cc:654] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:01.876886 14726 tablet_bootstrap.cc:654] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:01.877835 14727 log.cc:824] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:01.878103 14726 log.cc:824] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:01.879128 14725 tablet_bootstrap.cc:492] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: No bootstrap required, opened a new log
I20260928 02:50:01.879357 14725 ts_tablet_manager.cc:1397] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:50:01.879930 14727 tablet_bootstrap.cc:492] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: No bootstrap required, opened a new log
I20260928 02:50:01.879956 14726 tablet_bootstrap.cc:492] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: No bootstrap required, opened a new log
I20260928 02:50:01.880124 14727 ts_tablet_manager.cc:1397] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 02:50:01.880124 14726 ts_tablet_manager.cc:1397] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:50:01.882551 14725 raft_consensus.cc:348] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.882776 14725 raft_consensus.cc:374] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:01.882810 14725 raft_consensus.cc:729] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc5db68cedc438f9edccae03ab279a4, State: Initialized, Role: FOLLOWER
I20260928 02:50:01.883251 14725 consensus_queue.cc:260] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.883980 14574 heartbeater.cc:503] Master 127.28.248.190:46777 was elected leader, sending a full tablet report...
I20260928 02:50:01.884332 14725 ts_tablet_manager.cc:1428] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:50:01.884256 14726 raft_consensus.cc:348] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.884538 14726 raft_consensus.cc:374] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:01.884588 14726 raft_consensus.cc:729] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e649e924a489981c922d9d0208b7a, State: Initialized, Role: FOLLOWER
I20260928 02:50:01.884632 14727 raft_consensus.cc:348] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.884891 14727 raft_consensus.cc:374] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:01.884941 14727 raft_consensus.cc:729] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e44316354b54979b796a123a494880f, State: Initialized, Role: FOLLOWER
I20260928 02:50:01.885496 14726 consensus_queue.cc:260] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.885654 14727 consensus_queue.cc:260] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.886332 14440 heartbeater.cc:503] Master 127.28.248.190:46777 was elected leader, sending a full tablet report...
I20260928 02:50:01.886572 14727 ts_tablet_manager.cc:1428] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:50:01.886572 14726 ts_tablet_manager.cc:1428] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:50:01.886899 14708 heartbeater.cc:503] Master 127.28.248.190:46777 was elected leader, sending a full tablet report...
I20260928 02:50:01.900389 14732 raft_consensus.cc:482] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:01.900694 14732 raft_consensus.cc:504] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.901952 14732 leader_election.cc:288] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc5db68cedc438f9edccae03ab279a4 (127.28.248.130:45001), 6e44316354b54979b796a123a494880f (127.28.248.129:34545)
I20260928 02:50:01.906394 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.902168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:45001 (local address 127.28.248.131:54691)
0928 02:50:01.902288 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.902317 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.902417 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.902735 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.902738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.902755 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:01.902901 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.902911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.903892 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.903896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.905150 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.905160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.906008 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.906011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.906089 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.906108 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.906164 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.906241 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:50:01.906430 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.902287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54691 (local address 127.28.248.130:45001)
0928 02:50:01.902472 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.902476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.902494 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:50:01.902549 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.902554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.902620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.902702 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:01.903012 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.903770 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.905297 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.905886 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.906005 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.906064 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.906205 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.906296 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:50:01.906708 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.902282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34545 (local address 127.28.248.131:56277)
0928 02:50:01.902763 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.902776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:01.902879 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:01.903152 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:01.903155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:01.903171 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:01.903271 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.903276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.904263 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.904266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.905402 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:01.905409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.906287 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.906290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:01.906380 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.906402 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:50:01.906482 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:50:01.906552 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":10,"thread_start_us":125,"threads_started":1}
I20260928 02:50:01.907105 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.902468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:56277 (local address 127.28.248.129:34545)
0928 02:50:01.902587 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.902591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.902867 (+ 276us) server_negotiation.cc:407] Connection header received
0928 02:50:01.902976 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.902981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.903034 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.903130 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:01.903389 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.904145 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.905520 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.906168 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.906286 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.906328 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.906896 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.906973 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:50:01.907058 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aeb8f749b04623b33de3bfce75a695" candidate_uuid: "6a2e649e924a489981c922d9d0208b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc5db68cedc438f9edccae03ab279a4" is_pre_election: true
I20260928 02:50:01.907375 14528 raft_consensus.cc:2454] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e649e924a489981c922d9d0208b7a in term 0.
I20260928 02:50:01.907649 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aeb8f749b04623b33de3bfce75a695" candidate_uuid: "6a2e649e924a489981c922d9d0208b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e44316354b54979b796a123a494880f" is_pre_election: true
I20260928 02:50:01.907931 14394 raft_consensus.cc:2454] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2e649e924a489981c922d9d0208b7a in term 0.
I20260928 02:50:01.907918 14597 leader_election.cc:302] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e649e924a489981c922d9d0208b7a, bcc5db68cedc438f9edccae03ab279a4; no voters:
I20260928 02:50:01.908226 14732 raft_consensus.cc:2781] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:01.908365 14732 raft_consensus.cc:482] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:01.908432 14732 raft_consensus.cc:3037] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:01.909538 14732 raft_consensus.cc:504] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.909924 14732 leader_election.cc:288] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [CANDIDATE]: Term 1 election: Requested vote from peers bcc5db68cedc438f9edccae03ab279a4 (127.28.248.130:45001), 6e44316354b54979b796a123a494880f (127.28.248.129:34545)
I20260928 02:50:01.910317 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aeb8f749b04623b33de3bfce75a695" candidate_uuid: "6a2e649e924a489981c922d9d0208b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc5db68cedc438f9edccae03ab279a4"
I20260928 02:50:01.910298 14394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85aeb8f749b04623b33de3bfce75a695" candidate_uuid: "6a2e649e924a489981c922d9d0208b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e44316354b54979b796a123a494880f"
I20260928 02:50:01.910439 14394 raft_consensus.cc:3037] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:01.910439 14528 raft_consensus.cc:3037] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:01.911782 14528 raft_consensus.cc:2454] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e649e924a489981c922d9d0208b7a in term 1.
I20260928 02:50:01.911782 14394 raft_consensus.cc:2454] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e649e924a489981c922d9d0208b7a in term 1.
I20260928 02:50:01.912081 14597 leader_election.cc:302] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e649e924a489981c922d9d0208b7a, bcc5db68cedc438f9edccae03ab279a4; no voters:
I20260928 02:50:01.912309 14732 raft_consensus.cc:2781] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:01.912540 14732 raft_consensus.cc:686] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 1 LEADER]: Becoming Leader. State: Replica: 6a2e649e924a489981c922d9d0208b7a, State: Running, Role: LEADER
I20260928 02:50:01.912914 14732 consensus_queue.cc:237] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } }
I20260928 02:50:01.916344 14243 catalog_manager.cc:5495] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a2e649e924a489981c922d9d0208b7a (127.28.248.131). New cstate: current_term: 1 leader_uuid: "6a2e649e924a489981c922d9d0208b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e649e924a489981c922d9d0208b7a" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 34453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:01.941759 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.938103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.28.248.131:34453)
0928 02:50:01.938234 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:01.938237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:01.938253 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:50:01.938344 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:01.938346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:01.938404 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:01.938510 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:01.938722 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.939218 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.940057 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:01.940296 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:01.940393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:01.940756 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:01.941326 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:01.941427 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:01.941558 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:50:01.941616 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:50:01.948108 14528 raft_consensus.cc:1261] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 1 FOLLOWER]: Refusing update from remote peer 6a2e649e924a489981c922d9d0208b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:01.948108 14394 raft_consensus.cc:1261] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f [term 1 FOLLOWER]: Refusing update from remote peer 6a2e649e924a489981c922d9d0208b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:01.948916 14735 consensus_queue.cc:1035] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e44316354b54979b796a123a494880f" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:01.949126 14732 consensus_queue.cc:1035] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc5db68cedc438f9edccae03ab279a4" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:01.966514 14243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:01.966741 14243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:01.969333 14243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26937672808d4d39898d513f40e710cd: Sending DeleteTablet for 3 replicas of tablet 85aeb8f749b04623b33de3bfce75a695
I20260928 02:50:01.970329 14374 tablet_service.cc:1514] Processing DeleteTablet for tablet 85aeb8f749b04623b33de3bfce75a695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:01 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:56870
I20260928 02:50:01.970329 14642 tablet_service.cc:1514] Processing DeleteTablet for tablet 85aeb8f749b04623b33de3bfce75a695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:01 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:52448
I20260928 02:50:01.970474 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet 85aeb8f749b04623b33de3bfce75a695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:01 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:49082
I20260928 02:50:01.971100 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14311
I20260928 02:50:01.971256 14749 tablet_replica.cc:330] T 85aeb8f749b04623b33de3bfce75a695 P 6e44316354b54979b796a123a494880f: stopping tablet replica
I20260928 02:50:01.976958 14748 tablet_replica.cc:330] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: stopping tablet replica
I20260928 02:50:01.977384 14748 raft_consensus.cc:2229] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:01.977861 14750 tablet_replica.cc:330] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4: stopping tablet replica
I20260928 02:50:01.977964 14748 raft_consensus.cc:2258] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:01.978260 14750 raft_consensus.cc:2229] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 02:50:01.978631 14234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:34545 (error 108)
I20260928 02:50:01.978659 14750 raft_consensus.cc:2258] T 85aeb8f749b04623b33de3bfce75a695 P bcc5db68cedc438f9edccae03ab279a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:01.979466 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14445
I20260928 02:50:01.979912 14748 ts_tablet_manager.cc:1905] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:01.980938 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.980546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:34545 (local address 127.0.0.1:56874)
0928 02:50:01.980765 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.980839 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260928 02:50:01.981220 14234 catalog_manager.cc:4584] TS 6e44316354b54979b796a123a494880f (127.28.248.129:34545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85aeb8f749b04623b33de3bfce75a695: Network error: Client connection negotiation failed: client connection to 127.28.248.129:34545: connect: Connection refused (error 111)
I20260928 02:50:01.983446 14748 ts_tablet_manager.cc:1918] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:01.983572 14748 log.cc:1196] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/85aeb8f749b04623b33de3bfce75a695
I20260928 02:50:01.983980 14748 ts_tablet_manager.cc:1939] T 85aeb8f749b04623b33de3bfce75a695 P 6a2e649e924a489981c922d9d0208b7a: Deleting consensus metadata
I20260928 02:50:01.985571 14235 catalog_manager.cc:4841] TS 6a2e649e924a489981c922d9d0208b7a (127.28.248.131:34453): tablet 85aeb8f749b04623b33de3bfce75a695 (table test-table [id=fa35a42c78784383a43ce46ccfb9b4d5]) successfully deleted
I20260928 02:50:01.987511 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14579
I20260928 02:50:01.989303 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:01.987773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:45001 (local address 127.0.0.1:49092)
0928 02:50:01.989156 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:01.989219 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.130:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1269}
I20260928 02:50:01.994433 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14217
2026-09-28T02:50:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:02Z Disabled control of system clock
I20260928 02:50:02.027356 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38241
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:50:02.142098 14755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:02.142400 14755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:02.142438 14755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:02.146623 14755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:02.146782 14755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:02.146802 14755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:02.146821 14755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:02.146835 14755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:02.151765 14755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:38241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:02.152972 14755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:02.154424 14755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:02.163754 14761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.163756 14760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.163754 14763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:02.163988 14755 server_base.cc:1034] running on GCE node
I20260928 02:50:02.164837 14755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:02.166105 14755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:02.167273 14755 hybrid_clock.cc:648] HybridClock initialized: now 1790563802167260 us; error 29 us; skew 500 ppm
I20260928 02:50:02.169446 14755 webserver.cc:460] Webserver started at http://127.28.248.190:43759/ using document root <none> and password file <none>
I20260928 02:50:02.170027 14755 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:02.170089 14755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:02.170259 14755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:02.172372 14755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "e69eea8afa4743aabc6cdbcda442ea0a"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "7a9bcd4c7e8d9251e8c573789136af63"
server_key_iv: "6255f366436c5107ef40a76156ff4478"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.173044 14755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "e69eea8afa4743aabc6cdbcda442ea0a"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "7a9bcd4c7e8d9251e8c573789136af63"
server_key_iv: "6255f366436c5107ef40a76156ff4478"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.177081 14755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:02.179824 14769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.181258 14755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:02.181422 14755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "e69eea8afa4743aabc6cdbcda442ea0a"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "7a9bcd4c7e8d9251e8c573789136af63"
server_key_iv: "6255f366436c5107ef40a76156ff4478"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.181553 14755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:02.196589 14755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:02.197436 14755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:02.197652 14755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:02.204970 14755 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:38241
I20260928 02:50:02.204981 14821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:38241 every 8 connection(s)
I20260928 02:50:02.206019 14755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:02.209410 14822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:02.214207 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14755
I20260928 02:50:02.214373 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:02.214643 29666 external_mini_cluster.cc:1428] Setting key 50b1e76654a7b87bc2ef5952bb1c8549
I20260928 02:50:02.215560 14822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a: Bootstrap starting.
I20260928 02:50:02.218254 14822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:02.219099 14822 log.cc:824] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:02.221094 14822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a: No bootstrap required, opened a new log
I20260928 02:50:02.224061 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.216103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.28.248.190:38241)
0928 02:50:02.216814 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.216836 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.216875 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:50:02.217520 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.217538 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.217970 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.218320 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:02.218743 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.219795 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.220729 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.221226 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.221374 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.222243 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:02.222248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:02.222261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:02.222673 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:02.222736 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:02.222755 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:02.222834 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.223006 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.223303 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260928 02:50:02.225315 14822 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38241 } }
I20260928 02:50:02.225565 14822 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:02.225653 14822 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69eea8afa4743aabc6cdbcda442ea0a, State: Initialized, Role: FOLLOWER
I20260928 02:50:02.226086 14822 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38241 } }
I20260928 02:50:02.226245 14822 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:02.226336 14822 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:02.226461 14822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:02.227401 14822 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38241 } }
I20260928 02:50:02.227744 14822 leader_election.cc:302] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69eea8afa4743aabc6cdbcda442ea0a; no voters:
I20260928 02:50:02.228060 14822 leader_election.cc:288] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:02.228180 14827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:02.228447 14827 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [term 1 LEADER]: Becoming Leader. State: Replica: e69eea8afa4743aabc6cdbcda442ea0a, State: Running, Role: LEADER
I20260928 02:50:02.228847 14827 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38241 } }
I20260928 02:50:02.229332 14822 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:02.230532 14828 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38241 } } }
I20260928 02:50:02.230569 14829 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69eea8afa4743aabc6cdbcda442ea0a. Latest consensus state: current_term: 1 leader_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69eea8afa4743aabc6cdbcda442ea0a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38241 } } }
I20260928 02:50:02.230765 14829 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:02.230765 14828 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:02.231184 14837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:02.234690 14837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:02.240649 14837 catalog_manager.cc:1347] Generated new cluster ID: 0b714647172f4188bcc37039c38e3473
I20260928 02:50:02.240747 14837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:02.250931 14837 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:02.251893 14837 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:02.256222 14837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a: Generated new TSK 0
I20260928 02:50:02.257031 14837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:02.266229 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38241
--builtin_ntp_servers=127.28.248.148:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:50:02.381385 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:02.381660 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:02.381738 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:02.386027 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:02.386096 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:02.386205 14846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:50:02.391136 14846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:02.392369 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:02.393823 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:02.403913 14851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.403934 14852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.403952 14854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:02.404376 14846 server_base.cc:1034] running on GCE node
I20260928 02:50:02.404917 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:02.405726 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:02.406962 14846 hybrid_clock.cc:648] HybridClock initialized: now 1790563802406939 us; error 35 us; skew 500 ppm
I20260928 02:50:02.409348 14846 webserver.cc:460] Webserver started at http://127.28.248.129:37633/ using document root <none> and password file <none>
I20260928 02:50:02.410017 14846 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:02.410141 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:02.410424 14846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:02.413024 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "0b197e4a5fa343168aa1d71ddc4a1187"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "cb06c4575aaa4c18a9e7d0ceaf15cb46"
server_key_iv: "f2f6f455f8008adc6be2e5fffeb92502"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.413721 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "0b197e4a5fa343168aa1d71ddc4a1187"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "cb06c4575aaa4c18a9e7d0ceaf15cb46"
server_key_iv: "f2f6f455f8008adc6be2e5fffeb92502"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.418200 14846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:02.420902 14860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.422219 14846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:02.422371 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "0b197e4a5fa343168aa1d71ddc4a1187"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "cb06c4575aaa4c18a9e7d0ceaf15cb46"
server_key_iv: "f2f6f455f8008adc6be2e5fffeb92502"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.422551 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:02.446237 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:02.447079 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:02.447306 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:02.447983 14846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:02.449045 14846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:02.449115 14846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.449195 14846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:02.449239 14846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.458685 14846 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:46081
I20260928 02:50:02.458704 14973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:46081 every 8 connection(s)
I20260928 02:50:02.459640 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:02.463266 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14846
I20260928 02:50:02.463373 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:02.463668 29666 external_mini_cluster.cc:1428] Setting key e12cee7d7080663283cdfae4853fe16c
I20260928 02:50:02.465723 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38241
--builtin_ntp_servers=127.28.248.148:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:50:02.469602 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.461586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:49849 (local address 127.28.248.190:38241)
0928 02:50:02.461728 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.461732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.462783 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 02:50:02.464484 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.464488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.464544 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.464653 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:02.465687 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.466346 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.467517 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.467768 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.467885 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.469094 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:02.469098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:02.469101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:02.469121 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:02.469139 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:02.469147 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:02.469222 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.469396 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.469456 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:50:02.470507 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.461969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38241 (local address 127.28.248.129:49849)
0928 02:50:02.462625 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:02.462668 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:02.464265 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:02.464843 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:02.464850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:02.464940 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:02.465545 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.465559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.466488 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.466492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.467389 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.467398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.467931 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.467935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.468097 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.468798 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:02.468799 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:02.468869 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:02.468936 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:02.469235 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:02.469247 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:02.469259 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:50:02.469395 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:50:02.469690 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
I20260928 02:50:02.471894 14974 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38241
I20260928 02:50:02.472170 14974 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:02.472906 14974 heartbeater.cc:511] Master 127.28.248.190:38241 requested a full tablet report, sending...
I20260928 02:50:02.474833 14785 ts_manager.cc:194] Registered new tserver with Master: 0b197e4a5fa343168aa1d71ddc4a1187 (127.28.248.129:46081)
I20260928 02:50:02.476600 14785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.129:49849
W20260928 02:50:02.580103 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:02.580376 14977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:02.580418 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:02.584347 14977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:02.584410 14977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:02.584465 14977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:50:02.589303 14977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:02.590437 14977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:02.591818 14977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:02.601457 14985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.601457 14982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.601472 14983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:02.601599 14977 server_base.cc:1034] running on GCE node
I20260928 02:50:02.602221 14977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:02.602892 14977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:02.604096 14977 hybrid_clock.cc:648] HybridClock initialized: now 1790563802604083 us; error 33 us; skew 500 ppm
I20260928 02:50:02.606050 14977 webserver.cc:460] Webserver started at http://127.28.248.130:40311/ using document root <none> and password file <none>
I20260928 02:50:02.606648 14977 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:02.606743 14977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:02.606995 14977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:02.609127 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "6017fe7a9f4f47aebe3e11dade6d44fd"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "f18b11eb7f637a348ddc4340e8c222b3"
server_key_iv: "a396f90115226b3fffe1219c2b68a9ac"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.609704 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "6017fe7a9f4f47aebe3e11dade6d44fd"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "f18b11eb7f637a348ddc4340e8c222b3"
server_key_iv: "a396f90115226b3fffe1219c2b68a9ac"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.613534 14977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:50:02.616082 14991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.617395 14977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:02.617538 14977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "6017fe7a9f4f47aebe3e11dade6d44fd"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "f18b11eb7f637a348ddc4340e8c222b3"
server_key_iv: "a396f90115226b3fffe1219c2b68a9ac"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.617664 14977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:02.651577 14977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:02.652360 14977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:02.652578 14977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:02.653262 14977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:02.654279 14977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:02.654354 14977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.654425 14977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:02.654474 14977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.663897 14977 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:40601
I20260928 02:50:02.663916 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:40601 every 8 connection(s)
I20260928 02:50:02.664883 14977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:02.672899 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 14977
I20260928 02:50:02.673043 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:02.673345 29666 external_mini_cluster.cc:1428] Setting key dba13bc15549501ea7f6696ac2e80899
I20260928 02:50:02.674654 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.666826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:43957 (local address 127.28.248.190:38241)
0928 02:50:02.666974 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.666984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.668115 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 02:50:02.669602 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.669605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.669654 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.669732 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:02.670664 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.671189 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.672405 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.672758 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.672879 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.674169 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:02.674172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:02.674174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:02.674186 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:02.674205 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:02.674213 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:02.674271 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.674457 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.674510 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:50:02.675489 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38241
--builtin_ntp_servers=127.28.248.148:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:50:02.675721 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.667161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38241 (local address 127.28.248.130:43957)
0928 02:50:02.667947 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:02.667993 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:02.669398 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:02.669909 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:02.669916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:02.670010 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:02.670537 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.670551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.671333 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.671336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.672290 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.672299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.672918 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.672921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.673175 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.673881 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:02.673883 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:02.673926 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:02.673994 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:02.674298 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:02.674307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:02.674312 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:50:02.674449 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 02:50:02.674797 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260928 02:50:02.677066 15105 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38241
I20260928 02:50:02.677346 15105 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:02.678078 15105 heartbeater.cc:511] Master 127.28.248.190:38241 requested a full tablet report, sending...
I20260928 02:50:02.679311 14785 ts_manager.cc:194] Registered new tserver with Master: 6017fe7a9f4f47aebe3e11dade6d44fd (127.28.248.130:40601)
I20260928 02:50:02.680202 14785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.130:43957
W20260928 02:50:02.792701 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:02.792975 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:02.793021 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:02.797199 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:02.797268 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:02.797367 15108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:50:02.802520 15108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:02.803586 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:02.805162 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:02.815258 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.815294 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:02.815301 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:02.815447 15108 server_base.cc:1034] running on GCE node
I20260928 02:50:02.816016 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:02.816749 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:02.817907 15108 hybrid_clock.cc:648] HybridClock initialized: now 1790563802817883 us; error 42 us; skew 500 ppm
I20260928 02:50:02.819794 15108 webserver.cc:460] Webserver started at http://127.28.248.131:41105/ using document root <none> and password file <none>
I20260928 02:50:02.820400 15108 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:02.820464 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:02.820748 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:02.822937 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "318c13f2aef64e08bb0f34e0d41f12fa"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "45051ade29cc2cd36901a211b572e747"
server_key_iv: "47950cad344922f29d8918341d28ad51"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.823506 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "318c13f2aef64e08bb0f34e0d41f12fa"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "45051ade29cc2cd36901a211b572e747"
server_key_iv: "47950cad344922f29d8918341d28ad51"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.827423 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:50:02.830040 15122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.831434 15108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:50:02.831549 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "318c13f2aef64e08bb0f34e0d41f12fa"
format_stamp: "Formatted at 2026-09-28 02:50:02 on dist-test-slave-r238"
server_key: "45051ade29cc2cd36901a211b572e747"
server_key_iv: "47950cad344922f29d8918341d28ad51"
server_key_version: "encryptionkey@0"
I20260928 02:50:02.831655 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:02.854465 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:02.855423 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:02.855731 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:02.856508 15108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:02.857851 15108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:02.857971 15108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.858062 15108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:02.858120 15108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:02.870747 15108 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:42357
I20260928 02:50:02.870764 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:42357 every 8 connection(s)
I20260928 02:50:02.872170 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:02.872895 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15108
I20260928 02:50:02.873016 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:02.873345 29666 external_mini_cluster.cc:1428] Setting key 6f2f30f403e606f9432b883b9f58cd6d
I20260928 02:50:02.886440 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.875487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:46947 (local address 127.28.248.190:38241)
0928 02:50:02.875617 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.875621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.877155 (+ 1534us) server_negotiation.cc:407] Connection header received
0928 02:50:02.879362 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.879365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.879413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.879475 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:02.880871 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.881757 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.883709 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.884098 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.884212 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.885797 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:02.885801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:02.885803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:02.885818 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:02.885845 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:02.885854 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:02.885927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.886140 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.886222 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:02.888164 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.876066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:38241 (local address 127.28.248.131:46947)
0928 02:50:02.876981 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:02.877045 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:02.879075 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:02.879725 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:02.879738 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:02.879862 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:02.880698 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.880722 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.881904 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.881910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.883567 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.883583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.884298 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.884304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.884549 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.885413 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:02.885415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:02.885478 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:02.885584 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:02.885921 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:02.885934 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:02.885943 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:50:02.886749 (+ 806us) client_negotiation.cc:240] Negotiation successful
0928 02:50:02.887111 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":136,"threads_started":1}
I20260928 02:50:02.889338 15236 heartbeater.cc:347] Connected to a master server at 127.28.248.190:38241
I20260928 02:50:02.889582 15236 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:02.890105 15236 heartbeater.cc:511] Master 127.28.248.190:38241 requested a full tablet report, sending...
I20260928 02:50:02.891124 14785 ts_manager.cc:194] Registered new tserver with Master: 318c13f2aef64e08bb0f34e0d41f12fa (127.28.248.131:42357)
I20260928 02:50:02.891716 14785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.131:46947
I20260928 02:50:02.899766 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:02.905575 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.901955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.28.248.190:38241)
0928 02:50:02.902090 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.902096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.902223 (+ 127us) server_negotiation.cc:407] Connection header received
0928 02:50:02.902332 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.902335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.902378 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.902445 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:02.902693 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.903224 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.904033 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.904234 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.904317 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.905158 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:02.905160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:02.905162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:02.905174 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:02.905193 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:02.905200 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:02.905266 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.905342 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.905390 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:50:02.908829 14785 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:02.911932 14785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:02.926819 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.920589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:42357 (local address 127.0.0.1:39454)
0928 02:50:02.921656 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:02.921669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:02.921753 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:02.922425 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:02.922428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:02.922446 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:02.922558 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.922566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.923749 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.923755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.925206 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.925212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.926224 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.926231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.926350 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.926389 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:50:02.926506 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:50:02.926638 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":805,"thread_start_us":115,"threads_started":1}
I20260928 02:50:02.927030 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.920744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.28.248.129:46081)
0928 02:50:02.921128 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.921137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.921606 (+ 469us) server_negotiation.cc:407] Connection header received
0928 02:50:02.921907 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.921916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.922175 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.922376 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:02.922742 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.923901 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.925338 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.926195 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.926334 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.926536 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.926715 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.926825 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":137,"threads_started":1}
I20260928 02:50:02.927177 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.920885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.28.248.131:42357)
0928 02:50:02.922015 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.922027 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.922051 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:50:02.922136 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.922140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.922289 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.922449 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:02.922689 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.923645 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.925365 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.926037 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.926163 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.926340 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.926907 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.927011 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":150,"threads_started":1}
I20260928 02:50:02.927522 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.921387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40601 (local address 127.0.0.1:53242)
0928 02:50:02.922672 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:02.922685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:02.922766 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:02.923270 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:02.923273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:02.923292 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:02.923369 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.923374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.924755 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.924762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.925822 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.925834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.927129 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.927133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.927234 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.927257 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:50:02.927347 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:50:02.927426 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":84,"threads_started":1}
I20260928 02:50:02.928223 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.921718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.28.248.130:40601)
0928 02:50:02.922692 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:02.922712 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:02.922738 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:50:02.922910 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:02.922915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:02.923105 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:02.923275 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:02.923485 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.924556 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.925964 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.926799 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.926914 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.927063 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:02.927973 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 02:50:02.928072 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":167,"threads_started":1}
I20260928 02:50:02.927524 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:02.920582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:46081 (local address 127.0.0.1:34068)
0928 02:50:02.921299 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:02.921373 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:02.921652 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:02.922367 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:02.922373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:02.922396 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:02.922561 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.922571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.924024 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.924027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.925195 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:02.925207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:02.926464 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:02.926467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:02.926563 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:02.926592 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:50:02.926665 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:50:02.926745 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260928 02:50:02.931252 14908 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea3a0888a9846d3a6808b18720a99f3 (DEFAULT_TABLE table=test-table [id=b70f53fe25984704b4c82105de5d3f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:02.930879 15170 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea3a0888a9846d3a6808b18720a99f3 (DEFAULT_TABLE table=test-table [id=b70f53fe25984704b4c82105de5d3f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:02.932349 14908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3a0888a9846d3a6808b18720a99f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:02.932717 15170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3a0888a9846d3a6808b18720a99f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:02.934249 15039 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea3a0888a9846d3a6808b18720a99f3 (DEFAULT_TABLE table=test-table [id=b70f53fe25984704b4c82105de5d3f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:02.935146 15039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea3a0888a9846d3a6808b18720a99f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:02.940511 15254 tablet_bootstrap.cc:492] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: Bootstrap starting.
I20260928 02:50:02.940996 15253 tablet_bootstrap.cc:492] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Bootstrap starting.
I20260928 02:50:02.942108 15255 tablet_bootstrap.cc:492] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Bootstrap starting.
I20260928 02:50:02.943599 15254 tablet_bootstrap.cc:654] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:02.943599 15253 tablet_bootstrap.cc:654] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:02.944255 15255 tablet_bootstrap.cc:654] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:02.944509 15254 log.cc:824] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:02.944510 15253 log.cc:824] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:02.945107 15255 log.cc:824] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:02.946669 15254 tablet_bootstrap.cc:492] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: No bootstrap required, opened a new log
I20260928 02:50:02.946669 15253 tablet_bootstrap.cc:492] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: No bootstrap required, opened a new log
I20260928 02:50:02.946870 15253 ts_tablet_manager.cc:1397] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:50:02.946915 15254 ts_tablet_manager.cc:1397] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 02:50:02.947376 15255 tablet_bootstrap.cc:492] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: No bootstrap required, opened a new log
I20260928 02:50:02.947582 15255 ts_tablet_manager.cc:1397] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:50:02.951198 15255 raft_consensus.cc:348] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:02.951493 15255 raft_consensus.cc:374] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:02.951318 15253 raft_consensus.cc:348] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:02.951327 15254 raft_consensus.cc:348] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:02.951581 15255 raft_consensus.cc:729] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6017fe7a9f4f47aebe3e11dade6d44fd, State: Initialized, Role: FOLLOWER
I20260928 02:50:02.951601 15254 raft_consensus.cc:374] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:02.951653 15253 raft_consensus.cc:374] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:02.951658 15254 raft_consensus.cc:729] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b197e4a5fa343168aa1d71ddc4a1187, State: Initialized, Role: FOLLOWER
I20260928 02:50:02.951738 15253 raft_consensus.cc:729] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318c13f2aef64e08bb0f34e0d41f12fa, State: Initialized, Role: FOLLOWER
I20260928 02:50:02.952113 15255 consensus_queue.cc:260] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:02.952121 15253 consensus_queue.cc:260] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:02.952121 15254 consensus_queue.cc:260] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:02.953154 14974 heartbeater.cc:503] Master 127.28.248.190:38241 was elected leader, sending a full tablet report...
I20260928 02:50:02.953227 15254 ts_tablet_manager.cc:1428] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:50:02.953227 15253 ts_tablet_manager.cc:1428] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260928 02:50:02.953248 15255 ts_tablet_manager.cc:1428] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:50:02.953548 15105 heartbeater.cc:503] Master 127.28.248.190:38241 was elected leader, sending a full tablet report...
I20260928 02:50:02.953646 15236 heartbeater.cc:503] Master 127.28.248.190:38241 was elected leader, sending a full tablet report...
W20260928 02:50:02.960853 14975 tablet.cc:2369] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:03.080763 15260 raft_consensus.cc:482] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:03.080991 15260 raft_consensus.cc:504] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:03.082137 15260 leader_election.cc:288] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 318c13f2aef64e08bb0f34e0d41f12fa (127.28.248.131:42357), 6017fe7a9f4f47aebe3e11dade6d44fd (127.28.248.130:40601)
I20260928 02:50:03.088076 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:03.082708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:33399 (local address 127.28.248.130:40601)
0928 02:50:03.082847 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:03.082851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:03.083194 (+ 343us) server_negotiation.cc:407] Connection header received
0928 02:50:03.083564 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:03.083566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:03.083609 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:03.083667 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:03.084012 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.084773 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.086716 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.087342 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.087479 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:03.087531 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:03.087869 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:50:03.087938 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:03.088076 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:03.082837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:37179 (local address 127.28.248.131:42357)
0928 02:50:03.082962 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:03.082966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:03.083151 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:50:03.083323 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:03.083325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:03.083370 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:03.083450 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:03.083885 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.084773 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.086716 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.087354 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.087479 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:03.087531 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:03.087849 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 02:50:03.087933 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:50:03.088140 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:03.082579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:42357 (local address 127.28.248.129:37179)
0928 02:50:03.083036 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:03.083054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:03.083208 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:03.083544 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:03.083547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:03.083568 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:03.083749 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:03.083757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.084926 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.084930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:03.086569 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:03.086576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.087490 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.087494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:03.087670 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:03.087724 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:50:03.087808 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:50:03.087928 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":187,"thread_start_us":118,"threads_started":1}
I20260928 02:50:03.088140 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:03.082579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:40601 (local address 127.28.248.129:33399)
0928 02:50:03.083131 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:03.083143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:03.083215 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:03.083724 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:03.083727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:03.083749 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:03.083864 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:03.083874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.084926 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.084930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:03.086561 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:03.086576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.087490 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.087494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:03.087649 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:03.087735 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:50:03.087808 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:50:03.087932 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":306,"thread_start_us":90,"threads_started":1}
I20260928 02:50:03.088996 15190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3a0888a9846d3a6808b18720a99f3" candidate_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" is_pre_election: true
I20260928 02:50:03.089308 15059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3a0888a9846d3a6808b18720a99f3" candidate_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" is_pre_election: true
I20260928 02:50:03.089377 15190 raft_consensus.cc:2454] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b197e4a5fa343168aa1d71ddc4a1187 in term 0.
I20260928 02:50:03.089603 15059 raft_consensus.cc:2454] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b197e4a5fa343168aa1d71ddc4a1187 in term 0.
I20260928 02:50:03.090073 14863 leader_election.cc:302] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b197e4a5fa343168aa1d71ddc4a1187, 318c13f2aef64e08bb0f34e0d41f12fa; no voters:
I20260928 02:50:03.090407 15260 raft_consensus.cc:2781] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:03.090498 15260 raft_consensus.cc:482] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:03.090543 15260 raft_consensus.cc:3037] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:03.091658 15260 raft_consensus.cc:504] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:03.092051 15260 leader_election.cc:288] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [CANDIDATE]: Term 1 election: Requested vote from peers 318c13f2aef64e08bb0f34e0d41f12fa (127.28.248.131:42357), 6017fe7a9f4f47aebe3e11dade6d44fd (127.28.248.130:40601)
I20260928 02:50:03.092464 15190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3a0888a9846d3a6808b18720a99f3" candidate_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318c13f2aef64e08bb0f34e0d41f12fa"
I20260928 02:50:03.092535 15059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea3a0888a9846d3a6808b18720a99f3" candidate_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd"
I20260928 02:50:03.092589 15190 raft_consensus.cc:3037] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:03.092736 15059 raft_consensus.cc:3037] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:03.093772 15190 raft_consensus.cc:2454] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b197e4a5fa343168aa1d71ddc4a1187 in term 1.
I20260928 02:50:03.093890 15059 raft_consensus.cc:2454] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b197e4a5fa343168aa1d71ddc4a1187 in term 1.
I20260928 02:50:03.094141 14863 leader_election.cc:302] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b197e4a5fa343168aa1d71ddc4a1187, 318c13f2aef64e08bb0f34e0d41f12fa; no voters:
I20260928 02:50:03.094343 15260 raft_consensus.cc:2781] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:03.094558 15260 raft_consensus.cc:686] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [term 1 LEADER]: Becoming Leader. State: Replica: 0b197e4a5fa343168aa1d71ddc4a1187, State: Running, Role: LEADER
I20260928 02:50:03.094900 15260 consensus_queue.cc:237] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } }
I20260928 02:50:03.098163 14786 catalog_manager.cc:5495] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b197e4a5fa343168aa1d71ddc4a1187 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b197e4a5fa343168aa1d71ddc4a1187" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 46081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:03.122219 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:03.118857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.28.248.129:46081)
0928 02:50:03.118989 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:03.118992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:03.119005 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:03.119081 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:03.119084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:03.119138 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:03.119229 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:03.119456 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.119946 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.120876 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:03.121082 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:03.121162 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:03.121430 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:03.121902 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:03.121994 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:03.122064 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:50:03.122109 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:50:03.124404 15237 tablet.cc:2369] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:03.128450 15190 raft_consensus.cc:1261] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 1 FOLLOWER]: Refusing update from remote peer 0b197e4a5fa343168aa1d71ddc4a1187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:03.128450 15059 raft_consensus.cc:1261] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 1 FOLLOWER]: Refusing update from remote peer 0b197e4a5fa343168aa1d71ddc4a1187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:03.129212 15260 consensus_queue.cc:1035] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6017fe7a9f4f47aebe3e11dade6d44fd" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:03.129426 15264 consensus_queue.cc:1035] T 1ea3a0888a9846d3a6808b18720a99f3 P 0b197e4a5fa343168aa1d71ddc4a1187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "318c13f2aef64e08bb0f34e0d41f12fa" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 42357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:03.139125 15269 mvcc.cc:205] Tried to move back new op lower bound from 7334149337606742016 to 7334149337479557120. Current Snapshot: MvccSnapshot[applied={T|T < 7334149337606742016}]
I20260928 02:50:03.148689 14784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:03.148898 14784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:03.151643 14784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69eea8afa4743aabc6cdbcda442ea0a: Sending DeleteTablet for 3 replicas of tablet 1ea3a0888a9846d3a6808b18720a99f3
I20260928 02:50:03.152689 15170 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea3a0888a9846d3a6808b18720a99f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:03 UTC) from {username='slave'} at 127.0.0.1:39454
I20260928 02:50:03.152740 15039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea3a0888a9846d3a6808b18720a99f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:03 UTC) from {username='slave'} at 127.0.0.1:53242
I20260928 02:50:03.153165 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14846
I20260928 02:50:03.153237 14908 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea3a0888a9846d3a6808b18720a99f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:03 UTC) from {username='slave'} at 127.0.0.1:34068
I20260928 02:50:03.153385 15277 tablet_replica.cc:330] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: stopping tablet replica
I20260928 02:50:03.153796 15277 raft_consensus.cc:2229] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:03.153832 15278 tablet_replica.cc:330] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: stopping tablet replica
I20260928 02:50:03.154138 15277 raft_consensus.cc:2258] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:03.154150 15278 raft_consensus.cc:2229] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:03.154433 15278 raft_consensus.cc:2258] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:03.156095 15277 ts_tablet_manager.cc:1905] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:03.156742 15278 ts_tablet_manager.cc:1905] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:03.159073 15277 ts_tablet_manager.cc:1918] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:03.159188 15277 log.cc:1196] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/1ea3a0888a9846d3a6808b18720a99f3
I20260928 02:50:03.159461 15278 ts_tablet_manager.cc:1918] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:03.159565 15278 log.cc:1196] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/1ea3a0888a9846d3a6808b18720a99f3
I20260928 02:50:03.159561 15277 ts_tablet_manager.cc:1939] T 1ea3a0888a9846d3a6808b18720a99f3 P 318c13f2aef64e08bb0f34e0d41f12fa: Deleting consensus metadata
I20260928 02:50:03.159917 15278 ts_tablet_manager.cc:1939] T 1ea3a0888a9846d3a6808b18720a99f3 P 6017fe7a9f4f47aebe3e11dade6d44fd: Deleting consensus metadata
I20260928 02:50:03.161008 14772 catalog_manager.cc:4841] TS 318c13f2aef64e08bb0f34e0d41f12fa (127.28.248.131:42357): tablet 1ea3a0888a9846d3a6808b18720a99f3 (table test-table [id=b70f53fe25984704b4c82105de5d3f63]) successfully deleted
W20260928 02:50:03.161146 14771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:46081 (error 108)
I20260928 02:50:03.161362 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14977
I20260928 02:50:03.161656 14771 catalog_manager.cc:4841] TS 6017fe7a9f4f47aebe3e11dade6d44fd (127.28.248.130:40601): tablet 1ea3a0888a9846d3a6808b18720a99f3 (table test-table [id=b70f53fe25984704b4c82105de5d3f63]) successfully deleted
I20260928 02:50:03.162945 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:03.162539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:46081 (local address 127.0.0.1:34086)
0928 02:50:03.162737 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:03.162836 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:46081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260928 02:50:03.163152 14771 catalog_manager.cc:4584] TS 0b197e4a5fa343168aa1d71ddc4a1187 (127.28.248.129:46081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ea3a0888a9846d3a6808b18720a99f3: Network error: Client connection negotiation failed: client connection to 127.28.248.129:46081: connect: Connection refused (error 111)
I20260928 02:50:03.167466 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15108
I20260928 02:50:03.173091 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 14755
2026-09-28T02:50:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1176 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:03 dist-test-slave-r238 krb5kdc[15282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:03 dist-test-slave-r238 krb5kdc[15282](info): set up 2 sockets
Sep 28 02:50:03 dist-test-slave-r238 krb5kdc[15282](info): commencing operation
krb5kdc: starting...
W20260928 02:50:04.220747 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:04 dist-test-slave-r238 krb5kdc[15282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:04.347229 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:44415
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:44415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:04.460074 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:04.460383 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:04.460425 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:04.464637 15295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:04.464704 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:04.464723 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:04.464740 15295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:04.464756 15295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:04.470177 15295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:44415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:44415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:04.471307 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:04.472359 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:04.481052 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:04.481052 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:04.481204 15295 server_base.cc:1034] running on GCE node
W20260928 02:50:04.481415 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:04.481989 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:04.483165 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:04.484373 15295 hybrid_clock.cc:648] HybridClock initialized: now 1790563804484362 us; error 26 us; skew 500 ppm
Sep 28 02:50:04 dist-test-slave-r238 krb5kdc[15282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:04.487176 15295 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:04.488054 15295 webserver.cc:460] Webserver started at http://127.28.248.190:37821/ using document root <none> and password file <none>
I20260928 02:50:04.488588 15295 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:04.488722 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:04.488911 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:04.491043 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "7d4612291eff4932845dfb5daa27797c"
format_stamp: "Formatted at 2026-09-28 02:50:04 on dist-test-slave-r238"
server_key: "061db0df64c912fb88c390e0bacbba40"
server_key_iv: "3e0bc25df514f0fc0c0e8f4081e982eb"
server_key_version: "encryptionkey@0"
I20260928 02:50:04.491578 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "7d4612291eff4932845dfb5daa27797c"
format_stamp: "Formatted at 2026-09-28 02:50:04 on dist-test-slave-r238"
server_key: "061db0df64c912fb88c390e0bacbba40"
server_key_iv: "3e0bc25df514f0fc0c0e8f4081e982eb"
server_key_version: "encryptionkey@0"
I20260928 02:50:04.495463 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:50:04.497965 15310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:04.499231 15295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:04.499336 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "7d4612291eff4932845dfb5daa27797c"
format_stamp: "Formatted at 2026-09-28 02:50:04 on dist-test-slave-r238"
server_key: "061db0df64c912fb88c390e0bacbba40"
server_key_iv: "3e0bc25df514f0fc0c0e8f4081e982eb"
server_key_version: "encryptionkey@0"
I20260928 02:50:04.499428 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:04.513864 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:04.517045 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:04.517205 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:04.524827 15295 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:44415
I20260928 02:50:04.524848 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:44415 every 8 connection(s)
I20260928 02:50:04.525861 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:04.528710 15363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:04.533998 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15295
I20260928 02:50:04.534199 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:04.534485 29666 external_mini_cluster.cc:1428] Setting key 2c379af54ee338d1a2e9baca90e1906a
I20260928 02:50:04.534494 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c: Bootstrap starting.
I20260928 02:50:04.537154 15363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:04.537911 15363 log.cc:824] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:04.539839 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c: No bootstrap required, opened a new log
I20260928 02:50:04.543439 15363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4612291eff4932845dfb5daa27797c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 44415 } }
I20260928 02:50:04.543694 15363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:04.543776 15363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4612291eff4932845dfb5daa27797c, State: Initialized, Role: FOLLOWER
Sep 28 02:50:04 dist-test-slave-r238 krb5kdc[15282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:04.544296 15363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4612291eff4932845dfb5daa27797c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 44415 } }
I20260928 02:50:04.544441 15363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:04.544507 15363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:04.544623 15363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:04.545854 15363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4612291eff4932845dfb5daa27797c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 44415 } }
I20260928 02:50:04.546268 15363 leader_election.cc:302] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d4612291eff4932845dfb5daa27797c; no voters:
I20260928 02:50:04.546629 15363 leader_election.cc:288] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:04.546855 15368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:04.547137 15368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [term 1 LEADER]: Becoming Leader. State: Replica: 7d4612291eff4932845dfb5daa27797c, State: Running, Role: LEADER
I20260928 02:50:04.547547 15368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4612291eff4932845dfb5daa27797c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 44415 } }
I20260928 02:50:04.547950 15363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:04.548944 15368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4612291eff4932845dfb5daa27797c. Latest consensus state: current_term: 1 leader_uuid: "7d4612291eff4932845dfb5daa27797c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4612291eff4932845dfb5daa27797c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 44415 } } }
I20260928 02:50:04.549187 15368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:04.549388 15369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d4612291eff4932845dfb5daa27797c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4612291eff4932845dfb5daa27797c" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 44415 } } }
I20260928 02:50:04.549481 15369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:04.549858 15377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:04.551239 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:04.536006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.28.248.190:44415)
0928 02:50:04.536486 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:04.536503 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:04.536542 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:50:04.537428 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:04.537447 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:04.537806 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:04.538162 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:04.539084 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:04.540188 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:04.541190 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:04.541710 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:04.541904 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:04.544930 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:04.544936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:04.544954 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:04.544985 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:04.547118 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:04.547657 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:04.547663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:04.547668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:04.547822 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:04.548141 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:04.548144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:04.548146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:04.548693 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:04.549623 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:04.549821 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:50:04.550171 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260928 02:50:04.553599 15377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:04.559330 15377 catalog_manager.cc:1347] Generated new cluster ID: b1d0d4cbd3fe4077899da93aeadf359f
I20260928 02:50:04.559496 15377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:04.576036 15377 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:04.577086 15377 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:04.586177 15377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c: Generated new TSK 0
I20260928 02:50:04.587090 15377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:50:04.620422 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:44415
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:04.736953 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:04.737236 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:04.737282 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:04.741274 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:04.741335 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:04.741395 15389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:50:04.746928 15389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:04.748082 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:04.749260 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:04.759305 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:04.759320 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:04.759383 15389 server_base.cc:1034] running on GCE node
W20260928 02:50:04.759325 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:04.759922 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:04.760658 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:04.761865 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790563804761841 us; error 39 us; skew 500 ppm
Sep 28 02:50:04 dist-test-slave-r238 krb5kdc[15282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:04.764791 15389 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:50:04.765702 15389 webserver.cc:460] Webserver started at http://127.28.248.129:38925/ using document root <none> and password file <none>
I20260928 02:50:04.766319 15389 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:04.766417 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:04.766677 15389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:04.768920 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "40263b4e95e94e11bd9a607899aeaf7e"
format_stamp: "Formatted at 2026-09-28 02:50:04 on dist-test-slave-r238"
server_key: "2074ca160d1e04ee01be64b11364c361"
server_key_iv: "57c66876bfbcaf3dc98f37741cb8c521"
server_key_version: "encryptionkey@0"
I20260928 02:50:04.769517 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "40263b4e95e94e11bd9a607899aeaf7e"
format_stamp: "Formatted at 2026-09-28 02:50:04 on dist-test-slave-r238"
server_key: "2074ca160d1e04ee01be64b11364c361"
server_key_iv: "57c66876bfbcaf3dc98f37741cb8c521"
server_key_version: "encryptionkey@0"
I20260928 02:50:04.773564 15389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:04.776139 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:04.777554 15389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:04.777706 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "40263b4e95e94e11bd9a607899aeaf7e"
format_stamp: "Formatted at 2026-09-28 02:50:04 on dist-test-slave-r238"
server_key: "2074ca160d1e04ee01be64b11364c361"
server_key_iv: "57c66876bfbcaf3dc98f37741cb8c521"
server_key_version: "encryptionkey@0"
I20260928 02:50:04.777834 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:04.805688 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:04.809162 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:04.809391 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:04.810042 15389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:04.810943 15389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:04.811015 15389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:04.811091 15389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:04.811136 15389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:04.820372 15389 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:36133
I20260928 02:50:04.820391 15517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:36133 every 8 connection(s)
I20260928 02:50:04.821403 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:04.827560 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15389
I20260928 02:50:04.827725 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:04.828012 29666 external_mini_cluster.cc:1428] Setting key 0a5ee03c27342ec42b944e9b394ee94b
Sep 28 02:50:04 dist-test-slave-r238 krb5kdc[15282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:04.836742 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:04.823392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:57073 (local address 127.28.248.190:44415)
0928 02:50:04.823545 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:04.823548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:04.824415 (+ 867us) server_negotiation.cc:407] Connection header received
0928 02:50:04.825855 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:04.825858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:04.825919 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:04.826032 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:04.827198 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:04.827842 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:04.829267 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:04.829572 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:04.829670 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:04.832908 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:04.832912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:04.832914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:04.832938 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:04.834344 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:04.834992 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:04.834995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:04.834996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:04.835079 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:04.835521 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:04.835523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:04.835524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:04.835693 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:04.835765 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:04.836446 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 02:50:04.836546 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:50:04.837960 15520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:04.823775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:44415 (local address 127.28.248.129:57073)
0928 02:50:04.824257 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:04.824301 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:04.825628 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:04.826211 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:04.826218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:04.826581 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:04.827051 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:04.827064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:04.827972 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:04.827980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:04.829126 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:04.829136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:04.829729 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:04.829734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:04.829942 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:04.830572 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:04.830598 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:04.832718 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:04.834464 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:04.834473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:04.834491 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:04.834882 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:04.835192 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:04.835197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:04.835202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:04.835416 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:04.835802 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:04.835812 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:04.836178 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 02:50:04.836402 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 02:50:04.836812 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
I20260928 02:50:04.839284 15518 heartbeater.cc:347] Connected to a master server at 127.28.248.190:44415
I20260928 02:50:04.839494 15518 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:04.840068 15518 heartbeater.cc:511] Master 127.28.248.190:44415 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:50:04.841777 15327 ts_manager.cc:194] Registered new tserver with Master: 40263b4e95e94e11bd9a607899aeaf7e (127.28.248.129:36133)
I20260928 02:50:04.842947 15327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:57073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:50:04.859481 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:44415
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:04.973877 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:04.974262 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:04.974362 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:04.978839 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:04.978973 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:04.979076 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:50:04.985260 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:04.986481 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:04.987588 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:04.997849 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:04.997810 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:04.997872 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:04.998018 15523 server_base.cc:1034] running on GCE node
I20260928 02:50:04.998644 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:04.999347 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:05.000509 15523 hybrid_clock.cc:648] HybridClock initialized: now 1790563805000465 us; error 57 us; skew 500 ppm
Sep 28 02:50:04 dist-test-slave-r238 krb5kdc[15282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:05.003515 15523 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:50:05.004498 15523 webserver.cc:460] Webserver started at http://127.28.248.130:33293/ using document root <none> and password file <none>
I20260928 02:50:05.005121 15523 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:05.005191 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:05.005375 15523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:05.007567 15523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "1f062e764e584f4b97e288aa0ce4f341"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "3249bd8c7175f2f48af3e3481e247916"
server_key_iv: "0b3d8717a0a6aa220b5d87d6ae33f84b"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.008180 15523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "1f062e764e584f4b97e288aa0ce4f341"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "3249bd8c7175f2f48af3e3481e247916"
server_key_iv: "0b3d8717a0a6aa220b5d87d6ae33f84b"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.012113 15523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:50:05.014709 15538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.016130 15523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:05.016275 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "1f062e764e584f4b97e288aa0ce4f341"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "3249bd8c7175f2f48af3e3481e247916"
server_key_iv: "0b3d8717a0a6aa220b5d87d6ae33f84b"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.016404 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:05.037214 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:05.040961 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:05.041172 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:05.041790 15523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:05.042678 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:05.042752 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.042820 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:05.042871 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.052383 15523 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:45997
I20260928 02:50:05.052446 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:45997 every 8 connection(s)
I20260928 02:50:05.053450 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:05.056488 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15523
I20260928 02:50:05.056644 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:05.056962 29666 external_mini_cluster.cc:1428] Setting key 186397a65b5fd8dea0d9c962340e533c
Sep 28 02:50:05 dist-test-slave-r238 krb5kdc[15282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563804, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:50:05.070518 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.055465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:58201 (local address 127.28.248.190:44415)
0928 02:50:05.055626 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.055631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.056823 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 02:50:05.058534 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.058538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.058591 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.058682 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:05.059964 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.060653 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.062186 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.062617 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.062753 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.066196 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:05.066202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:05.066207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:05.066251 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:05.068095 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:05.068769 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:05.068772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:05.068773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:05.068854 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:05.069264 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:05.069271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:05.069273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:05.069516 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:05.069605 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.070263 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.070391 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:50:05.071704 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.055959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:44415 (local address 127.28.248.130:58201)
0928 02:50:05.056512 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.056600 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.058262 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.058901 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.058909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.059280 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:05.059798 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.059813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.060799 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.060805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.062044 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.062054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.062880 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.062886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.063095 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.063794 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:05.063822 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:05.065945 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:05.068277 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:05.068290 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:05.068313 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:05.068666 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:05.068959 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:05.068961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:05.068962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:05.069157 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:05.069620 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:05.069626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:05.069951 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.070146 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.070482 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":160,"threads_started":1}
I20260928 02:50:05.073374 15652 heartbeater.cc:347] Connected to a master server at 127.28.248.190:44415
I20260928 02:50:05.073662 15652 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:05.074215 15652 heartbeater.cc:511] Master 127.28.248.190:44415 requested a full tablet report, sending...
I20260928 02:50:05.075423 15327 ts_manager.cc:194] Registered new tserver with Master: 1f062e764e584f4b97e288aa0ce4f341 (127.28.248.130:45997)
I20260928 02:50:05.076336 15327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:58201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:50:05.091688 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:44415
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:05.214887 15657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:05.215174 15657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:05.215222 15657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:05.219234 15657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:05.219298 15657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:05.219358 15657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:50:05.224957 15657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:44415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:05.226022 15657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:05.227016 15657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:05.236814 15665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:05.236833 15662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:05.236833 15663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:05.236894 15657 server_base.cc:1034] running on GCE node
I20260928 02:50:05.237602 15657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:05.238279 15657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:05.239454 15657 hybrid_clock.cc:648] HybridClock initialized: now 1790563805239438 us; error 34 us; skew 500 ppm
Sep 28 02:50:05 dist-test-slave-r238 krb5kdc[15282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563805, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:05.242535 15657 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:50:05.243503 15657 webserver.cc:460] Webserver started at http://127.28.248.131:37607/ using document root <none> and password file <none>
I20260928 02:50:05.244104 15657 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:05.244196 15657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:05.244421 15657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:05.246676 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "1c54a8ff33ab436bb84763fb50c271c9"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "5d8b0bd62f1f44878ed3bd9cb0d29582"
server_key_iv: "609eb6c9bed083f8ad7693d491943985"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.247301 15657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "1c54a8ff33ab436bb84763fb50c271c9"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "5d8b0bd62f1f44878ed3bd9cb0d29582"
server_key_iv: "609eb6c9bed083f8ad7693d491943985"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.251794 15657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 02:50:05.254678 15672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.256125 15657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:05.256299 15657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "1c54a8ff33ab436bb84763fb50c271c9"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "5d8b0bd62f1f44878ed3bd9cb0d29582"
server_key_iv: "609eb6c9bed083f8ad7693d491943985"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.256438 15657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:05.285940 15657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:05.289844 15657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:05.290083 15657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:05.290748 15657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:05.291709 15657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:05.291790 15657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.291869 15657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:05.291924 15657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.301839 15657 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:44229
I20260928 02:50:05.301860 15785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:44229 every 8 connection(s)
I20260928 02:50:05.302901 15657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:05.309567 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15657
I20260928 02:50:05.309720 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:05.310009 29666 external_mini_cluster.cc:1428] Setting key 77a121fc05356eada4f997b69af8bfa8
Sep 28 02:50:05 dist-test-slave-r238 krb5kdc[15282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563805, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:05.317286 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.304910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:54711 (local address 127.28.248.190:44415)
0928 02:50:05.305086 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.305089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.305979 (+ 890us) server_negotiation.cc:407] Connection header received
0928 02:50:05.307356 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.307360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.307417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.307525 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:05.308904 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.309453 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.310772 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.310993 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.311107 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.313818 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:05.313821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:05.313824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:05.313848 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:05.315203 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:05.315715 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:05.315718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:05.315720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:05.315802 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:05.316135 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:05.316140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:05.316142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:05.316333 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:05.316441 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.317006 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.317125 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:50:05.318135 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.305354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:44415 (local address 127.28.248.131:54711)
0928 02:50:05.305820 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.305864 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.307129 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.307704 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.307711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.308115 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:05.308721 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.308747 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.309584 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.309588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.310622 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.310632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.311148 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.311152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.311334 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.311991 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:05.312007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:05.313600 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:05.315324 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:05.315330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:05.315347 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:05.315603 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:05.315895 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:05.315899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:05.315901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:05.316041 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:05.316438 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:05.316446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:05.316769 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.316956 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.317301 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260928 02:50:05.319406 15786 heartbeater.cc:347] Connected to a master server at 127.28.248.190:44415
I20260928 02:50:05.319641 15786 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:05.320209 15786 heartbeater.cc:511] Master 127.28.248.190:44415 requested a full tablet report, sending...
I20260928 02:50:05.321254 15327 ts_manager.cc:194] Registered new tserver with Master: 1c54a8ff33ab436bb84763fb50c271c9 (127.28.248.131:44229)
I20260928 02:50:05.321825 15327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:54711
I20260928 02:50:05.323911 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:05.334021 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.325970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.28.248.190:44415)
0928 02:50:05.326125 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.326130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.326318 (+ 188us) server_negotiation.cc:407] Connection header received
0928 02:50:05.326474 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.326477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.326526 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.326610 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:05.327306 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.327864 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.328770 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.328991 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.329100 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.330656 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:05.330660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:05.330663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:05.330698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:05.332102 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:05.332676 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:05.332683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:05.332685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:05.332767 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:05.333104 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:05.333107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:05.333108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:05.333319 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:05.333476 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.333745 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.333854 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:05.337189 15327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:05.340278 15327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:05.355988 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.349312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.28.248.129:36133)
0928 02:50:05.349623 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.349631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.350388 (+ 757us) server_negotiation.cc:407] Connection header received
0928 02:50:05.350706 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.350712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.350927 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.351024 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:05.351562 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.353010 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.354319 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.355192 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.355336 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.355502 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.355707 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.355826 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":129,"threads_started":1}
I20260928 02:50:05.356298 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.350359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.28.248.130:45997)
0928 02:50:05.350779 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.350790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.350808 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:50:05.350871 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.350875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.351017 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.351173 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:05.351597 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.352777 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.354576 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.355362 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.355473 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.355582 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.356059 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.356170 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":134,"threads_started":1}
I20260928 02:50:05.356411 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.349704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.28.248.131:44229)
0928 02:50:05.350073 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.350092 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.350731 (+ 639us) server_negotiation.cc:407] Connection header received
0928 02:50:05.350813 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.350819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.351017 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.351173 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:05.351615 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.353500 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.354855 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.355527 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.355660 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.355795 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.356169 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.356281 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260928 02:50:05.355981 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.349165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:36133 (local address 127.0.0.1:35924)
0928 02:50:05.350234 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.350288 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.350505 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.351191 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.351193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.351228 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:05.351425 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.351434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.353116 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.353119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.354184 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.354194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.355333 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.355336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.355461 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.355511 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.355662 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.355795 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":57,"spinlock_wait_cycles":5248,"thread_start_us":102,"threads_started":1}
I20260928 02:50:05.355989 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.350192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:45997 (local address 127.0.0.1:54730)
0928 02:50:05.350616 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.350631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.350716 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.351186 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.351191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.351230 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:05.351438 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.351447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.352926 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.352929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.354442 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.354452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.355580 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.355584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.355687 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.355710 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.355782 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.355861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":44,"spinlock_wait_cycles":15872,"thread_start_us":100,"threads_started":1}
I20260928 02:50:05.357023 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.349173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:44229 (local address 127.0.0.1:56582)
0928 02:50:05.350495 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.350511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.350602 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.351366 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.351369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.351398 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:05.351482 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.351490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.353656 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.353660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.354739 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.354746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.355855 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.355858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.355942 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.355963 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.356033 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.356094 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":631,"thread_start_us":142,"threads_started":1}
I20260928 02:50:05.359985 15586 tablet_service.cc:1467] Processing CreateTablet for tablet b22ced74a4fb49d6bd11c22619fc3047 (DEFAULT_TABLE table=test-table [id=2c5a53445fda4980977703737482e220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:05.359985 15720 tablet_service.cc:1467] Processing CreateTablet for tablet b22ced74a4fb49d6bd11c22619fc3047 (DEFAULT_TABLE table=test-table [id=2c5a53445fda4980977703737482e220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:05.361203 15720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22ced74a4fb49d6bd11c22619fc3047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:05.361203 15586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22ced74a4fb49d6bd11c22619fc3047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:05.362675 15452 tablet_service.cc:1467] Processing CreateTablet for tablet b22ced74a4fb49d6bd11c22619fc3047 (DEFAULT_TABLE table=test-table [id=2c5a53445fda4980977703737482e220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:05.363547 15452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22ced74a4fb49d6bd11c22619fc3047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:05.368892 15803 tablet_bootstrap.cc:492] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Bootstrap starting.
I20260928 02:50:05.369691 15804 tablet_bootstrap.cc:492] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Bootstrap starting.
I20260928 02:50:05.370360 15805 tablet_bootstrap.cc:492] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: Bootstrap starting.
I20260928 02:50:05.371508 15803 tablet_bootstrap.cc:654] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:05.371758 15804 tablet_bootstrap.cc:654] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:05.372633 15805 tablet_bootstrap.cc:654] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:05.373154 15804 log.cc:824] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:05.373196 15803 log.cc:824] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:05.373466 15805 log.cc:824] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:05.375270 15805 tablet_bootstrap.cc:492] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: No bootstrap required, opened a new log
I20260928 02:50:05.375339 15804 tablet_bootstrap.cc:492] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: No bootstrap required, opened a new log
I20260928 02:50:05.375329 15803 tablet_bootstrap.cc:492] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: No bootstrap required, opened a new log
I20260928 02:50:05.375506 15805 ts_tablet_manager.cc:1397] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:50:05.375527 15804 ts_tablet_manager.cc:1397] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 02:50:05.375525 15803 ts_tablet_manager.cc:1397] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:50:05.379060 15805 raft_consensus.cc:348] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.379417 15805 raft_consensus.cc:374] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:05.379493 15805 raft_consensus.cc:729] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40263b4e95e94e11bd9a607899aeaf7e, State: Initialized, Role: FOLLOWER
I20260928 02:50:05.379451 15804 raft_consensus.cc:348] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.379451 15803 raft_consensus.cc:348] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.379705 15803 raft_consensus.cc:374] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:05.379733 15804 raft_consensus.cc:374] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:05.379799 15803 raft_consensus.cc:729] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f062e764e584f4b97e288aa0ce4f341, State: Initialized, Role: FOLLOWER
I20260928 02:50:05.379806 15804 raft_consensus.cc:729] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c54a8ff33ab436bb84763fb50c271c9, State: Initialized, Role: FOLLOWER
I20260928 02:50:05.380069 15805 consensus_queue.cc:260] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.380292 15803 consensus_queue.cc:260] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.380291 15804 consensus_queue.cc:260] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.380846 15518 heartbeater.cc:503] Master 127.28.248.190:44415 was elected leader, sending a full tablet report...
I20260928 02:50:05.381240 15805 ts_tablet_manager.cc:1428] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:50:05.381270 15803 ts_tablet_manager.cc:1428] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 02:50:05.381330 15804 ts_tablet_manager.cc:1428] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:50:05.381592 15786 heartbeater.cc:503] Master 127.28.248.190:44415 was elected leader, sending a full tablet report...
I20260928 02:50:05.381944 15652 heartbeater.cc:503] Master 127.28.248.190:44415 was elected leader, sending a full tablet report...
W20260928 02:50:05.554025 15787 tablet.cc:2369] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:05.554658 15653 tablet.cc:2369] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:05.572907 15519 tablet.cc:2369] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:05.580200 15811 raft_consensus.cc:482] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:05.580461 15811 raft_consensus.cc:504] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.581846 15811 leader_election.cc:288] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40263b4e95e94e11bd9a607899aeaf7e (127.28.248.129:36133), 1c54a8ff33ab436bb84763fb50c271c9 (127.28.248.131:44229)
I20260928 02:50:05.587352 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.582257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:46771 (local address 127.28.248.131:44229)
0928 02:50:05.582408 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.582413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.582632 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:50:05.582738 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.582741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.582796 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.582887 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:05.583350 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.584195 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.586180 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.586713 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.586815 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.586860 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.587111 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.587228 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:50:05.587353 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.582137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:44229 (local address 127.28.248.130:46771)
0928 02:50:05.582491 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.582508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.582637 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.582919 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.582922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.582962 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:05.583194 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.583202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.584311 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.584314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.585984 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.585996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.586837 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.586841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.586955 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.586978 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.587055 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.587148 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":239,"spinlock_wait_cycles":9600,"thread_start_us":101,"threads_started":1}
I20260928 02:50:05.587388 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.582265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:33421 (local address 127.28.248.129:36133)
0928 02:50:05.582408 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.582413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.582640 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:50:05.582738 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.582741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.582796 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.582887 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:05.583350 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.584145 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.586144 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.586695 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.586816 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.586868 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.587191 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.587280 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:50:05.587353 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.582137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:36133 (local address 127.28.248.130:33421)
0928 02:50:05.582530 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.582544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:05.582637 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:05.582919 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:05.582922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:05.582961 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:05.583201 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.583207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.584311 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.584314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.585987 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:05.585996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.586954 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.586958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:05.587058 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.587077 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:50:05.587138 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:50:05.587188 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":264,"spinlock_wait_cycles":38656,"thread_start_us":77,"threads_started":1}
I20260928 02:50:05.588255 15472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22ced74a4fb49d6bd11c22619fc3047" candidate_uuid: "1f062e764e584f4b97e288aa0ce4f341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40263b4e95e94e11bd9a607899aeaf7e" is_pre_election: true
I20260928 02:50:05.588253 15740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22ced74a4fb49d6bd11c22619fc3047" candidate_uuid: "1f062e764e584f4b97e288aa0ce4f341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c54a8ff33ab436bb84763fb50c271c9" is_pre_election: true
I20260928 02:50:05.588632 15472 raft_consensus.cc:2454] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f062e764e584f4b97e288aa0ce4f341 in term 0.
I20260928 02:50:05.588634 15740 raft_consensus.cc:2454] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f062e764e584f4b97e288aa0ce4f341 in term 0.
I20260928 02:50:05.589195 15539 leader_election.cc:302] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c54a8ff33ab436bb84763fb50c271c9, 1f062e764e584f4b97e288aa0ce4f341, 40263b4e95e94e11bd9a607899aeaf7e; no voters:
I20260928 02:50:05.589548 15811 raft_consensus.cc:2781] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:05.589654 15811 raft_consensus.cc:482] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:05.589735 15811 raft_consensus.cc:3037] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:05.590817 15811 raft_consensus.cc:504] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.591184 15811 leader_election.cc:288] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [CANDIDATE]: Term 1 election: Requested vote from peers 40263b4e95e94e11bd9a607899aeaf7e (127.28.248.129:36133), 1c54a8ff33ab436bb84763fb50c271c9 (127.28.248.131:44229)
I20260928 02:50:05.591593 15740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22ced74a4fb49d6bd11c22619fc3047" candidate_uuid: "1f062e764e584f4b97e288aa0ce4f341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c54a8ff33ab436bb84763fb50c271c9"
I20260928 02:50:05.591591 15472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22ced74a4fb49d6bd11c22619fc3047" candidate_uuid: "1f062e764e584f4b97e288aa0ce4f341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40263b4e95e94e11bd9a607899aeaf7e"
I20260928 02:50:05.591727 15472 raft_consensus.cc:3037] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:05.591727 15740 raft_consensus.cc:3037] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:05.593089 15472 raft_consensus.cc:2454] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f062e764e584f4b97e288aa0ce4f341 in term 1.
I20260928 02:50:05.593089 15740 raft_consensus.cc:2454] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f062e764e584f4b97e288aa0ce4f341 in term 1.
I20260928 02:50:05.593443 15542 leader_election.cc:302] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c54a8ff33ab436bb84763fb50c271c9, 1f062e764e584f4b97e288aa0ce4f341, 40263b4e95e94e11bd9a607899aeaf7e; no voters:
I20260928 02:50:05.593677 15811 raft_consensus.cc:2781] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:05.593865 15811 raft_consensus.cc:686] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 1 LEADER]: Becoming Leader. State: Replica: 1f062e764e584f4b97e288aa0ce4f341, State: Running, Role: LEADER
I20260928 02:50:05.594198 15811 consensus_queue.cc:237] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } }
I20260928 02:50:05.597642 15326 catalog_manager.cc:5495] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f062e764e584f4b97e288aa0ce4f341 (127.28.248.130). New cstate: current_term: 1 leader_uuid: "1f062e764e584f4b97e288aa0ce4f341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f062e764e584f4b97e288aa0ce4f341" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 45997 } health_report { overall_health: HEALTHY } } }
I20260928 02:50:05.652830 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.648951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.28.248.130:45997)
0928 02:50:05.649104 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.649109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.649126 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:50:05.649263 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.649266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.649322 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.649459 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:05.649727 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.650254 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.651122 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.651362 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.651486 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.651807 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:05.652239 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:05.652410 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.652495 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.652537 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:50:05.659127 15740 raft_consensus.cc:1261] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 1 FOLLOWER]: Refusing update from remote peer 1f062e764e584f4b97e288aa0ce4f341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:05.659127 15472 raft_consensus.cc:1261] T b22ced74a4fb49d6bd11c22619fc3047 P 40263b4e95e94e11bd9a607899aeaf7e [term 1 FOLLOWER]: Refusing update from remote peer 1f062e764e584f4b97e288aa0ce4f341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:05.659859 15811 consensus_queue.cc:1035] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40263b4e95e94e11bd9a607899aeaf7e" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:05.660162 15814 consensus_queue.cc:1035] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c54a8ff33ab436bb84763fb50c271c9" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 44229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:05.667817 15818 mvcc.cc:205] Tried to move back new op lower bound from 7334149347971641344 to 7334149347716886528. Current Snapshot: MvccSnapshot[applied={T|T < 7334149347971641344}]
I20260928 02:50:05.668002 15819 mvcc.cc:205] Tried to move back new op lower bound from 7334149347971641344 to 7334149347716886528. Current Snapshot: MvccSnapshot[applied={T|T < 7334149347971641344}]
I20260928 02:50:05.668409 15817 mvcc.cc:205] Tried to move back new op lower bound from 7334149347971641344 to 7334149347716886528. Current Snapshot: MvccSnapshot[applied={T|T < 7334149347971641344}]
I20260928 02:50:05.678984 15326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:05.679167 15326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:05.681900 15326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d4612291eff4932845dfb5daa27797c: Sending DeleteTablet for 3 replicas of tablet b22ced74a4fb49d6bd11c22619fc3047
I20260928 02:50:05.682824 15452 tablet_service.cc:1514] Processing DeleteTablet for tablet b22ced74a4fb49d6bd11c22619fc3047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:05 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:35924
I20260928 02:50:05.682987 15720 tablet_service.cc:1514] Processing DeleteTablet for tablet b22ced74a4fb49d6bd11c22619fc3047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:05 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:56582
I20260928 02:50:05.683648 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15389
I20260928 02:50:05.683756 15586 tablet_service.cc:1514] Processing DeleteTablet for tablet b22ced74a4fb49d6bd11c22619fc3047 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:05 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:54730
I20260928 02:50:05.684687 15829 tablet_replica.cc:330] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: stopping tablet replica
I20260928 02:50:05.685027 15829 raft_consensus.cc:2229] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:05.685487 15829 raft_consensus.cc:2258] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:05.685887 15828 tablet_replica.cc:330] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: stopping tablet replica
I20260928 02:50:05.686203 15828 raft_consensus.cc:2229] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:05.686535 15828 raft_consensus.cc:2258] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:05.688002 15829 ts_tablet_manager.cc:1905] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:05.688730 15828 ts_tablet_manager.cc:1905] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:05.691071 15829 ts_tablet_manager.cc:1918] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:05.691267 15829 log.cc:1196] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/b22ced74a4fb49d6bd11c22619fc3047
I20260928 02:50:05.691708 15829 ts_tablet_manager.cc:1939] T b22ced74a4fb49d6bd11c22619fc3047 P 1f062e764e584f4b97e288aa0ce4f341: Deleting consensus metadata
I20260928 02:50:05.692207 15828 ts_tablet_manager.cc:1918] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:05.692384 15828 log.cc:1196] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/b22ced74a4fb49d6bd11c22619fc3047
I20260928 02:50:05.692822 15828 ts_tablet_manager.cc:1939] T b22ced74a4fb49d6bd11c22619fc3047 P 1c54a8ff33ab436bb84763fb50c271c9: Deleting consensus metadata
I20260928 02:50:05.693125 15311 catalog_manager.cc:4841] TS 1f062e764e584f4b97e288aa0ce4f341 (127.28.248.130:45997): tablet b22ced74a4fb49d6bd11c22619fc3047 (table test-table [id=2c5a53445fda4980977703737482e220]) successfully deleted
W20260928 02:50:05.693233 15314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:36133 (error 108)
I20260928 02:50:05.694344 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.694011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:36133 (local address 127.0.0.1:35940)
0928 02:50:05.694161 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:05.694238 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:36133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:50:05.694401 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15523
I20260928 02:50:05.694482 15311 catalog_manager.cc:4841] TS 1c54a8ff33ab436bb84763fb50c271c9 (127.28.248.131:44229): tablet b22ced74a4fb49d6bd11c22619fc3047 (table test-table [id=2c5a53445fda4980977703737482e220]) successfully deleted
W20260928 02:50:05.694629 15314 catalog_manager.cc:4584] TS 40263b4e95e94e11bd9a607899aeaf7e (127.28.248.129:36133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b22ced74a4fb49d6bd11c22619fc3047: Network error: Client connection negotiation failed: client connection to 127.28.248.129:36133: connect: Connection refused (error 111)
I20260928 02:50:05.700242 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15657
I20260928 02:50:05.706677 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15295
2026-09-28T02:50:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2534 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:05Z Disabled control of system clock
I20260928 02:50:05.737998 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45939
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:45939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:50:05.851596 15834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:05.851922 15834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:05.851961 15834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:05.856031 15834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:05.856106 15834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:05.856122 15834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:05.856140 15834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:05.856156 15834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:05.861141 15834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:45939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:45939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:05.862273 15834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:05.863629 15834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:05.872761 15834 server_base.cc:1034] running on GCE node
W20260928 02:50:05.872651 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:05.872660 15839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:05.872664 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:05.873577 15834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:05.874733 15834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:05.875939 15834 hybrid_clock.cc:648] HybridClock initialized: now 1790563805875905 us; error 51 us; skew 500 ppm
I20260928 02:50:05.877861 15834 webserver.cc:460] Webserver started at http://127.28.248.190:41619/ using document root <none> and password file <none>
I20260928 02:50:05.878407 15834 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:05.878465 15834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:05.878647 15834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:05.880756 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "632c2ae1677f48e49f6c4adce4ff322d"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "964bc98908501226e5aacf8c2f2f9973"
server_key_iv: "46bd83ecdbb5f7e17ddab344106cb396"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.881304 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "632c2ae1677f48e49f6c4adce4ff322d"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "964bc98908501226e5aacf8c2f2f9973"
server_key_iv: "46bd83ecdbb5f7e17ddab344106cb396"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.885176 15834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:50:05.887589 15848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:05.889019 15834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:05.889160 15834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "632c2ae1677f48e49f6c4adce4ff322d"
format_stamp: "Formatted at 2026-09-28 02:50:05 on dist-test-slave-r238"
server_key: "964bc98908501226e5aacf8c2f2f9973"
server_key_iv: "46bd83ecdbb5f7e17ddab344106cb396"
server_key_version: "encryptionkey@0"
I20260928 02:50:05.889286 15834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:05.925318 15834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:05.926118 15834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:05.926352 15834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:05.933686 15834 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:45939
I20260928 02:50:05.933691 15900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:45939 every 8 connection(s)
I20260928 02:50:05.934748 15834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:05.938118 15901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:05.943907 15901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d: Bootstrap starting.
I20260928 02:50:05.944368 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15834
I20260928 02:50:05.944511 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:05.944854 29666 external_mini_cluster.cc:1428] Setting key bc61e3a3227a380ccf80e5a60505b359
I20260928 02:50:05.946841 15901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:05.947624 15901 log.cc:824] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:05.949842 15901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d: No bootstrap required, opened a new log
I20260928 02:50:05.953403 15901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632c2ae1677f48e49f6c4adce4ff322d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45939 } }
I20260928 02:50:05.953670 15901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:05.953744 15901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632c2ae1677f48e49f6c4adce4ff322d, State: Initialized, Role: FOLLOWER
I20260928 02:50:05.954203 15901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632c2ae1677f48e49f6c4adce4ff322d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45939 } }
I20260928 02:50:05.954385 15901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:05.954478 15901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:05.954608 15901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:05.954854 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:05.946273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.28.248.190:45939)
0928 02:50:05.947103 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:05.947120 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:05.947156 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:50:05.947874 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:05.947893 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:05.948374 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:05.948817 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:05.949256 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.950288 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.951179 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:05.951688 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:05.951915 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:05.952860 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:05.952870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:05.952884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:05.953280 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:05.953336 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:05.953357 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:05.953469 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:05.953584 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:50:05.953899 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":157,"threads_started":1}
I20260928 02:50:05.955832 15901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632c2ae1677f48e49f6c4adce4ff322d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45939 } }
I20260928 02:50:05.956189 15901 leader_election.cc:302] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 632c2ae1677f48e49f6c4adce4ff322d; no voters:
I20260928 02:50:05.956511 15901 leader_election.cc:288] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:05.956642 15906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:05.956915 15906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [term 1 LEADER]: Becoming Leader. State: Replica: 632c2ae1677f48e49f6c4adce4ff322d, State: Running, Role: LEADER
I20260928 02:50:05.957283 15906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632c2ae1677f48e49f6c4adce4ff322d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45939 } }
I20260928 02:50:05.957881 15901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:05.958563 15906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632c2ae1677f48e49f6c4adce4ff322d. Latest consensus state: current_term: 1 leader_uuid: "632c2ae1677f48e49f6c4adce4ff322d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632c2ae1677f48e49f6c4adce4ff322d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45939 } } }
I20260928 02:50:05.958789 15906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:05.959005 15907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "632c2ae1677f48e49f6c4adce4ff322d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632c2ae1677f48e49f6c4adce4ff322d" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 45939 } } }
I20260928 02:50:05.959154 15907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:05.959244 15915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:05.962116 15915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:05.968199 15915 catalog_manager.cc:1347] Generated new cluster ID: 92b607d6ab254ff2823fe47984714b32
I20260928 02:50:05.968281 15915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:05.979559 15915 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:05.980912 15915 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:05.987924 15915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d: Generated new TSK 0
I20260928 02:50:05.988754 15915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:05.998239 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45939
--builtin_ntp_servers=127.28.248.148:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:50:06.117722 15925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:06.118049 15925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:06.118096 15925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:06.122565 15925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:06.122644 15925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:06.122705 15925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:50:06.127741 15925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:06.128942 15925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:06.130298 15925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:06.140569 15925 server_base.cc:1034] running on GCE node
W20260928 02:50:06.140517 15930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:06.140506 15931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:06.140506 15933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:06.141438 15925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:06.142091 15925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:06.143273 15925 hybrid_clock.cc:648] HybridClock initialized: now 1790563806143225 us; error 56 us; skew 500 ppm
I20260928 02:50:06.145582 15925 webserver.cc:460] Webserver started at http://127.28.248.129:34309/ using document root <none> and password file <none>
I20260928 02:50:06.146137 15925 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:06.146200 15925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:06.146394 15925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:06.148552 15925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "40ddd3fa47ea4180b96cc06b4181aa8d"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "a650c02fe0d8b49d5a2de7bc82e27010"
server_key_iv: "f989a1c3a3f9b7f92063839c72d84d7e"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.149156 15925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "40ddd3fa47ea4180b96cc06b4181aa8d"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "a650c02fe0d8b49d5a2de7bc82e27010"
server_key_iv: "f989a1c3a3f9b7f92063839c72d84d7e"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.152912 15925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:06.155424 15939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.156780 15925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:06.156886 15925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "40ddd3fa47ea4180b96cc06b4181aa8d"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "a650c02fe0d8b49d5a2de7bc82e27010"
server_key_iv: "f989a1c3a3f9b7f92063839c72d84d7e"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.156980 15925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:06.166472 15925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:06.167107 15925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:06.167269 15925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:06.167783 15925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:06.168744 15925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:06.168795 15925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.168838 15925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:06.168853 15925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.178591 15925 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:35513
I20260928 02:50:06.178617 16052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:35513 every 8 connection(s)
I20260928 02:50:06.179563 15925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:06.185129 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 15925
I20260928 02:50:06.185282 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:06.185580 29666 external_mini_cluster.cc:1428] Setting key 8c7aea05caf29eb77007cd96a8c85a3a
I20260928 02:50:06.187573 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45939
--builtin_ntp_servers=127.28.248.148:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:50:06.189878 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.181695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:54251 (local address 127.28.248.190:45939)
0928 02:50:06.181943 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.181946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.182928 (+ 982us) server_negotiation.cc:407] Connection header received
0928 02:50:06.184387 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.184391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.184450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.184554 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:06.185602 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.186287 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.187608 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.187908 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.188023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.189337 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:06.189340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:06.189342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:06.189355 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:06.189375 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:06.189382 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:06.189449 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.189651 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.189707 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 02:50:06.191179 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.182072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45939 (local address 127.28.248.129:54251)
0928 02:50:06.182779 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.182821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.184142 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.184764 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.184771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.184862 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:06.185459 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.185476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.186418 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.186421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.187481 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.187492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.188067 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.188071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.188253 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.188991 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:06.188994 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:06.189060 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:06.189156 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:06.189471 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:06.189482 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:06.189488 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.189658 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.190056 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260928 02:50:06.192512 16053 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45939
I20260928 02:50:06.192849 16053 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:06.193563 16053 heartbeater.cc:511] Master 127.28.248.190:45939 requested a full tablet report, sending...
I20260928 02:50:06.195322 15865 ts_manager.cc:194] Registered new tserver with Master: 40ddd3fa47ea4180b96cc06b4181aa8d (127.28.248.129:35513)
I20260928 02:50:06.196509 15865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.129:54251
W20260928 02:50:06.303524 16056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:06.303989 16056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:06.304086 16056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:06.308519 16056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:06.308681 16056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:06.308794 16056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:50:06.314109 16056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:06.315294 16056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:06.316787 16056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:06.327276 16064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:06.327276 16061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:06.327456 16056 server_base.cc:1034] running on GCE node
W20260928 02:50:06.327276 16062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:06.327934 16056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:06.328745 16056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:06.329921 16056 hybrid_clock.cc:648] HybridClock initialized: now 1790563806329898 us; error 36 us; skew 500 ppm
I20260928 02:50:06.331895 16056 webserver.cc:460] Webserver started at http://127.28.248.130:34845/ using document root <none> and password file <none>
I20260928 02:50:06.332515 16056 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:06.332579 16056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:06.332901 16056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:06.335121 16056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "e8e07da990e042c29c423d40900cce5e"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "ffdfe533533a1c9c77e80781fe5326b1"
server_key_iv: "2952535d0a0c8f9010aa2bb7bfeec5c1"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.335721 16056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "e8e07da990e042c29c423d40900cce5e"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "ffdfe533533a1c9c77e80781fe5326b1"
server_key_iv: "2952535d0a0c8f9010aa2bb7bfeec5c1"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.339773 16056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:50:06.342487 16070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.343871 16056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:06.344033 16056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "e8e07da990e042c29c423d40900cce5e"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "ffdfe533533a1c9c77e80781fe5326b1"
server_key_iv: "2952535d0a0c8f9010aa2bb7bfeec5c1"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.344162 16056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:06.359833 16056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:06.360683 16056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:06.360906 16056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:06.361527 16056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:06.362536 16056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:06.362609 16056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.362682 16056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:06.362733 16056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.371659 16056 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:33773
I20260928 02:50:06.371677 16183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:33773 every 8 connection(s)
I20260928 02:50:06.372731 16056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:06.375161 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16056
I20260928 02:50:06.375301 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:06.375666 29666 external_mini_cluster.cc:1428] Setting key d5f5cf19791036b65dc22dabd4790c9b
I20260928 02:50:06.377770 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45939
--builtin_ntp_servers=127.28.248.148:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:50:06.383917 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.374688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:40551 (local address 127.28.248.190:45939)
0928 02:50:06.374839 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.374843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.376280 (+ 1437us) server_negotiation.cc:407] Connection header received
0928 02:50:06.377899 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.377903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.377967 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.378054 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:06.378990 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.379613 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.381317 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.381750 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.381879 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.383211 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:06.383217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:06.383222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:06.383244 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:06.383273 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:06.383286 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:06.383378 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.383542 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.383631 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:06.384819 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.375212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45939 (local address 127.28.248.130:40551)
0928 02:50:06.376088 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.376147 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.377667 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.378227 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.378234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.378324 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:06.378839 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.378852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.379747 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.379753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.381166 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.381179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.381934 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.381937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.382108 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.382860 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:06.382862 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:06.382907 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:06.382979 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:06.383392 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:06.383398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:06.383403 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.383504 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.383804 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":173,"threads_started":1}
I20260928 02:50:06.386332 16184 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45939
I20260928 02:50:06.386621 16184 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:06.387202 16184 heartbeater.cc:511] Master 127.28.248.190:45939 requested a full tablet report, sending...
I20260928 02:50:06.388434 15865 ts_manager.cc:194] Registered new tserver with Master: e8e07da990e042c29c423d40900cce5e (127.28.248.130:33773)
I20260928 02:50:06.389418 15865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.130:40551
W20260928 02:50:06.495867 16187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:06.496165 16187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:06.496206 16187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:06.500272 16187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:06.500344 16187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:06.500403 16187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:50:06.505465 16187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:06.506611 16187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:06.508005 16187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:06.518442 16187 server_base.cc:1034] running on GCE node
W20260928 02:50:06.518340 16192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:06.518388 16195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:06.518340 16193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:06.519174 16187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:06.519907 16187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:06.521099 16187 hybrid_clock.cc:648] HybridClock initialized: now 1790563806521065 us; error 52 us; skew 500 ppm
I20260928 02:50:06.523196 16187 webserver.cc:460] Webserver started at http://127.28.248.131:34661/ using document root <none> and password file <none>
I20260928 02:50:06.523811 16187 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:06.523921 16187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:06.524161 16187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:06.526408 16187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "91814264987743dc926bba9bd76d60ba"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "ea511c5b694e8398abf398e0ce5659c3"
server_key_iv: "c7d7d1603e448cbf0ed5fadb5748765c"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.526999 16187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "91814264987743dc926bba9bd76d60ba"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "ea511c5b694e8398abf398e0ce5659c3"
server_key_iv: "c7d7d1603e448cbf0ed5fadb5748765c"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.530973 16187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:50:06.533660 16201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.535037 16187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:06.535183 16187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "91814264987743dc926bba9bd76d60ba"
format_stamp: "Formatted at 2026-09-28 02:50:06 on dist-test-slave-r238"
server_key: "ea511c5b694e8398abf398e0ce5659c3"
server_key_iv: "c7d7d1603e448cbf0ed5fadb5748765c"
server_key_version: "encryptionkey@0"
I20260928 02:50:06.535310 16187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:06.568115 16187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:06.568933 16187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:06.569149 16187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:06.569777 16187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:06.570789 16187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:06.570875 16187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.570950 16187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:06.570992 16187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:06.579936 16187 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:39083
I20260928 02:50:06.579955 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:39083 every 8 connection(s)
I20260928 02:50:06.580993 16187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:06.586055 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16187
I20260928 02:50:06.586217 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:06.586493 29666 external_mini_cluster.cc:1428] Setting key c07b36714364a9b281d9b2cae47c73e9
I20260928 02:50:06.591172 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.582860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:50425 (local address 127.28.248.190:45939)
0928 02:50:06.583038 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.583043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.584162 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 02:50:06.585693 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.585696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.585760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.585866 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:06.587011 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.587595 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.589244 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.589472 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.589561 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.590794 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:06.590796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:06.590798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:06.590810 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:06.590826 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:06.590833 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:06.590888 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.591001 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.591056 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:50:06.592098 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.583324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:45939 (local address 127.28.248.131:50425)
0928 02:50:06.584000 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.584042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.585393 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.586053 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.586061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.586149 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:06.586818 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.586836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.587740 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.587743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.589110 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.589119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.589650 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.589653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.589824 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.590540 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:06.590542 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:06.590584 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:06.590653 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:06.590887 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:06.590892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:06.590895 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.590984 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.591244 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260928 02:50:06.593391 16315 heartbeater.cc:347] Connected to a master server at 127.28.248.190:45939
I20260928 02:50:06.593670 16315 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:06.594306 16315 heartbeater.cc:511] Master 127.28.248.190:45939 requested a full tablet report, sending...
I20260928 02:50:06.595415 15865 ts_manager.cc:194] Registered new tserver with Master: 91814264987743dc926bba9bd76d60ba (127.28.248.131:39083)
I20260928 02:50:06.596097 15865 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.131:50425
I20260928 02:50:06.600656 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:06.606328 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.602593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.28.248.190:45939)
0928 02:50:06.602736 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.602740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.602908 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:50:06.603006 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.603008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.603048 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.603117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:06.603370 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.603871 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.604742 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.604962 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.605038 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.605922 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:06.605925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:06.605927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:06.605938 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:06.605954 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:06.605961 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:06.606052 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.606174 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.606206 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:50:06.609239 15865 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:06.612471 15865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:06.628028 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.621798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.28.248.129:35513)
0928 02:50:06.622544 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.622558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.622581 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:50:06.622661 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.622666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.622903 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.623205 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:06.623506 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.624707 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.626513 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.627183 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.627307 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.627502 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.627721 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.627835 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":156,"threads_started":1}
I20260928 02:50:06.628021 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.621712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:39083 (local address 127.0.0.1:52826)
0928 02:50:06.622424 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.622439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.622509 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.623673 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.623676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.623694 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:06.623778 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.623785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.624950 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.624954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.626385 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.626396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.627358 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.627361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.627509 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.627552 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.627657 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.627860 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":108,"thread_start_us":124,"threads_started":1}
I20260928 02:50:06.628435 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.622720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.28.248.131:39083)
0928 02:50:06.623204 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.623215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.623238 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:50:06.623307 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.623312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.623522 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.623696 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:06.623917 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.624832 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.626527 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.627208 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.627935 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.628109 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.628160 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.628267 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":152,"threads_started":1}
I20260928 02:50:06.628464 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.621537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:35513 (local address 127.0.0.1:49562)
0928 02:50:06.622103 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.622155 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.622291 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.623205 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.623213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.623239 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:06.623374 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.623380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.624816 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.624825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.626377 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.626386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.627331 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.627337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.627524 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.627552 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.627666 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.627860 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":394,"thread_start_us":79,"threads_started":1}
I20260928 02:50:06.628703 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.623036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.28.248.130:33773)
0928 02:50:06.623523 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.623535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.623574 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:50:06.623647 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.623652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.623848 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.624000 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:06.624190 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.625328 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.626900 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.627663 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.627785 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.627970 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.628399 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.628501 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":172,"threads_started":1}
I20260928 02:50:06.628991 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.622310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33773 (local address 127.0.0.1:46982)
0928 02:50:06.622520 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.622532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.622604 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.623983 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.623985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.624003 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:06.624074 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.624080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.625463 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.625466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.626778 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.626788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.628040 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.628047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.628158 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.628193 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.628267 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.628329 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":181,"thread_start_us":100,"threads_started":1}
I20260928 02:50:06.631421 15987 tablet_service.cc:1467] Processing CreateTablet for tablet 40139cd10d384445a70161142afb4fd5 (DEFAULT_TABLE table=test-table [id=d0a917b46187434d8e8a667b0cea1b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:06.631542 16118 tablet_service.cc:1467] Processing CreateTablet for tablet 40139cd10d384445a70161142afb4fd5 (DEFAULT_TABLE table=test-table [id=d0a917b46187434d8e8a667b0cea1b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:06.631542 16249 tablet_service.cc:1467] Processing CreateTablet for tablet 40139cd10d384445a70161142afb4fd5 (DEFAULT_TABLE table=test-table [id=d0a917b46187434d8e8a667b0cea1b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:06.632437 15987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40139cd10d384445a70161142afb4fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:06.632540 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40139cd10d384445a70161142afb4fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:06.632541 16249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40139cd10d384445a70161142afb4fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:06.638765 16332 tablet_bootstrap.cc:492] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: Bootstrap starting.
I20260928 02:50:06.640499 16334 tablet_bootstrap.cc:492] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Bootstrap starting.
I20260928 02:50:06.641324 16332 tablet_bootstrap.cc:654] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:06.642184 16332 log.cc:824] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:06.642531 16334 tablet_bootstrap.cc:654] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:06.643000 16333 tablet_bootstrap.cc:492] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Bootstrap starting.
I20260928 02:50:06.643355 16334 log.cc:824] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:06.644388 16332 tablet_bootstrap.cc:492] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: No bootstrap required, opened a new log
I20260928 02:50:06.644598 16332 ts_tablet_manager.cc:1397] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:50:06.645095 16333 tablet_bootstrap.cc:654] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:06.645486 16334 tablet_bootstrap.cc:492] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: No bootstrap required, opened a new log
I20260928 02:50:06.645668 16334 ts_tablet_manager.cc:1397] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:50:06.645865 16333 log.cc:824] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:06.647603 16333 tablet_bootstrap.cc:492] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: No bootstrap required, opened a new log
I20260928 02:50:06.647799 16333 ts_tablet_manager.cc:1397] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:50:06.649612 16332 raft_consensus.cc:348] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.649920 16332 raft_consensus.cc:374] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:06.649988 16332 raft_consensus.cc:729] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ddd3fa47ea4180b96cc06b4181aa8d, State: Initialized, Role: FOLLOWER
I20260928 02:50:06.650144 16334 raft_consensus.cc:348] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.650386 16334 raft_consensus.cc:374] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:06.650445 16334 raft_consensus.cc:729] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91814264987743dc926bba9bd76d60ba, State: Initialized, Role: FOLLOWER
I20260928 02:50:06.650460 16332 consensus_queue.cc:260] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.650924 16334 consensus_queue.cc:260] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.651377 16053 heartbeater.cc:503] Master 127.28.248.190:45939 was elected leader, sending a full tablet report...
I20260928 02:50:06.651654 16332 ts_tablet_manager.cc:1428] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 02:50:06.651820 16334 ts_tablet_manager.cc:1428] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 02:50:06.651952 16333 raft_consensus.cc:348] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.652143 16315 heartbeater.cc:503] Master 127.28.248.190:45939 was elected leader, sending a full tablet report...
I20260928 02:50:06.652184 16333 raft_consensus.cc:374] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:06.652241 16333 raft_consensus.cc:729] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e07da990e042c29c423d40900cce5e, State: Initialized, Role: FOLLOWER
I20260928 02:50:06.652724 16333 consensus_queue.cc:260] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.653388 16184 heartbeater.cc:503] Master 127.28.248.190:45939 was elected leader, sending a full tablet report...
I20260928 02:50:06.653754 16333 ts_tablet_manager.cc:1428] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260928 02:50:06.680876 16054 tablet.cc:2369] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:06.787339 16338 raft_consensus.cc:482] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:06.787572 16338 raft_consensus.cc:504] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.788727 16338 leader_election.cc:288] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91814264987743dc926bba9bd76d60ba (127.28.248.131:39083), e8e07da990e042c29c423d40900cce5e (127.28.248.130:33773)
I20260928 02:50:06.792742 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.789109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:44439 (local address 127.28.248.131:39083)
0928 02:50:06.789256 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.789260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.789429 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:50:06.789572 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.789576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.789627 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.789812 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:06.790094 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.790596 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.791746 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.792176 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.792299 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.792351 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.792525 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.792589 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:50:06.792761 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.789018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:39083 (local address 127.28.248.129:44439)
0928 02:50:06.789298 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.789319 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.789446 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.789825 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.789828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.789847 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:06.789974 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.789980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.790725 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.790731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.791622 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.791631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.792290 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.792294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.792400 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.792427 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.792510 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.792580 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":49,"thread_start_us":120,"threads_started":1}
I20260928 02:50:06.793109 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.789625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:46539 (local address 127.28.248.130:33773)
0928 02:50:06.789781 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.789786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.789901 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:50:06.790050 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.790053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.790094 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.790179 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:06.790450 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.790940 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.791995 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.792436 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.792572 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.792640 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.792894 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.792977 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:50:06.793403 16269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40139cd10d384445a70161142afb4fd5" candidate_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91814264987743dc926bba9bd76d60ba" is_pre_election: true
I20260928 02:50:06.793524 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.789458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:33773 (local address 127.28.248.129:46539)
0928 02:50:06.789809 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.789822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:06.789941 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:06.790190 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:06.790192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:06.790207 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:06.790342 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.790347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.791053 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.791058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.791869 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:06.791877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.792562 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.792566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:06.792688 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.792709 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:50:06.793257 (+ 548us) client_negotiation.cc:240] Negotiation successful
0928 02:50:06.793389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":74,"threads_started":1}
I20260928 02:50:06.793743 16269 raft_consensus.cc:2454] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ddd3fa47ea4180b96cc06b4181aa8d in term 0.
I20260928 02:50:06.794178 16138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40139cd10d384445a70161142afb4fd5" candidate_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e07da990e042c29c423d40900cce5e" is_pre_election: true
I20260928 02:50:06.794260 15942 leader_election.cc:302] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ddd3fa47ea4180b96cc06b4181aa8d, 91814264987743dc926bba9bd76d60ba; no voters:
I20260928 02:50:06.794478 16138 raft_consensus.cc:2454] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ddd3fa47ea4180b96cc06b4181aa8d in term 0.
I20260928 02:50:06.794526 16338 raft_consensus.cc:2781] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:06.794647 16338 raft_consensus.cc:482] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:06.794704 16338 raft_consensus.cc:3037] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:06.795785 16338 raft_consensus.cc:504] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.796089 16338 leader_election.cc:288] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [CANDIDATE]: Term 1 election: Requested vote from peers 91814264987743dc926bba9bd76d60ba (127.28.248.131:39083), e8e07da990e042c29c423d40900cce5e (127.28.248.130:33773)
I20260928 02:50:06.796458 16269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40139cd10d384445a70161142afb4fd5" candidate_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91814264987743dc926bba9bd76d60ba"
I20260928 02:50:06.796515 16138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40139cd10d384445a70161142afb4fd5" candidate_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e07da990e042c29c423d40900cce5e"
I20260928 02:50:06.796648 16138 raft_consensus.cc:3037] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:06.796638 16269 raft_consensus.cc:3037] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:06.797966 16138 raft_consensus.cc:2454] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ddd3fa47ea4180b96cc06b4181aa8d in term 1.
I20260928 02:50:06.797966 16269 raft_consensus.cc:2454] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ddd3fa47ea4180b96cc06b4181aa8d in term 1.
I20260928 02:50:06.798257 15942 leader_election.cc:302] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ddd3fa47ea4180b96cc06b4181aa8d, 91814264987743dc926bba9bd76d60ba; no voters:
I20260928 02:50:06.798415 16338 raft_consensus.cc:2781] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:06.798631 16338 raft_consensus.cc:686] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 1 LEADER]: Becoming Leader. State: Replica: 40ddd3fa47ea4180b96cc06b4181aa8d, State: Running, Role: LEADER
I20260928 02:50:06.798969 16338 consensus_queue.cc:237] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } }
I20260928 02:50:06.802366 15865 catalog_manager.cc:5495] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 40ddd3fa47ea4180b96cc06b4181aa8d (127.28.248.129). New cstate: current_term: 1 leader_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ddd3fa47ea4180b96cc06b4181aa8d" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:06.825345 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.821443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.28.248.129:35513)
0928 02:50:06.821601 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:06.821605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:06.821620 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:50:06.821795 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:06.821798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:06.821863 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:06.821984 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:06.822272 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.822789 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.823676 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:06.823905 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:06.824013 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:06.824359 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:06.824852 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:06.825014 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:06.825141 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:50:06.825184 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:50:06.831730 16269 raft_consensus.cc:1261] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 1 FOLLOWER]: Refusing update from remote peer 40ddd3fa47ea4180b96cc06b4181aa8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:06.831730 16138 raft_consensus.cc:1261] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 1 FOLLOWER]: Refusing update from remote peer 40ddd3fa47ea4180b96cc06b4181aa8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 02:50:06.831962 16316 tablet.cc:2369] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:06.832541 16338 consensus_queue.cc:1035] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e07da990e042c29c423d40900cce5e" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 33773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:06.832757 16343 consensus_queue.cc:1035] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "91814264987743dc926bba9bd76d60ba" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:06.840816 16348 mvcc.cc:205] Tried to move back new op lower bound from 7334149352774852608 to 7334149352651587584. Current Snapshot: MvccSnapshot[applied={T|T < 7334149352774852608}]
I20260928 02:50:06.850140 15862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:06.850346 15862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:06.853080 15862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 632c2ae1677f48e49f6c4adce4ff322d: Sending DeleteTablet for 3 replicas of tablet 40139cd10d384445a70161142afb4fd5
I20260928 02:50:06.853909 15987 tablet_service.cc:1514] Processing DeleteTablet for tablet 40139cd10d384445a70161142afb4fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:06 UTC) from {username='slave'} at 127.0.0.1:49562
I20260928 02:50:06.853973 16249 tablet_service.cc:1514] Processing DeleteTablet for tablet 40139cd10d384445a70161142afb4fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:06 UTC) from {username='slave'} at 127.0.0.1:52826
I20260928 02:50:06.854162 16118 tablet_service.cc:1514] Processing DeleteTablet for tablet 40139cd10d384445a70161142afb4fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:06 UTC) from {username='slave'} at 127.0.0.1:46982
I20260928 02:50:06.854629 16358 tablet_replica.cc:330] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: stopping tablet replica
I20260928 02:50:06.854678 16357 tablet_replica.cc:330] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d: stopping tablet replica
I20260928 02:50:06.855031 16357 raft_consensus.cc:2229] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:06.855033 16358 raft_consensus.cc:2229] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:06.855185 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15925
I20260928 02:50:06.855420 16357 raft_consensus.cc:2258] T 40139cd10d384445a70161142afb4fd5 P 40ddd3fa47ea4180b96cc06b4181aa8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:06.855575 16358 raft_consensus.cc:2258] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:06.855800 16356 tablet_replica.cc:330] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: stopping tablet replica
I20260928 02:50:06.856144 16356 raft_consensus.cc:2229] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:06.856463 16356 raft_consensus.cc:2258] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:06.857259 16358 ts_tablet_manager.cc:1905] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:06.860316 16358 ts_tablet_manager.cc:1918] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:06.860417 16358 log.cc:1196] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/40139cd10d384445a70161142afb4fd5
I20260928 02:50:06.860532 16356 ts_tablet_manager.cc:1905] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:06.860773 16358 ts_tablet_manager.cc:1939] T 40139cd10d384445a70161142afb4fd5 P e8e07da990e042c29c423d40900cce5e: Deleting consensus metadata
I20260928 02:50:06.862085 15851 catalog_manager.cc:4841] TS e8e07da990e042c29c423d40900cce5e (127.28.248.130:33773): tablet 40139cd10d384445a70161142afb4fd5 (table test-table [id=d0a917b46187434d8e8a667b0cea1b31]) successfully deleted
I20260928 02:50:06.863049 16356 ts_tablet_manager.cc:1918] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:06.863142 16356 log.cc:1196] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/40139cd10d384445a70161142afb4fd5
I20260928 02:50:06.863437 16356 ts_tablet_manager.cc:1939] T 40139cd10d384445a70161142afb4fd5 P 91814264987743dc926bba9bd76d60ba: Deleting consensus metadata
W20260928 02:50:06.863482 15852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:35513 (error 108)
I20260928 02:50:06.863591 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16056
I20260928 02:50:06.864769 15851 catalog_manager.cc:4841] TS 91814264987743dc926bba9bd76d60ba (127.28.248.131:39083): tablet 40139cd10d384445a70161142afb4fd5 (table test-table [id=d0a917b46187434d8e8a667b0cea1b31]) successfully deleted
I20260928 02:50:06.869634 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16187
I20260928 02:50:06.869668 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:06.869242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:35513 (local address 127.0.0.1:49594)
0928 02:50:06.869481 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:06.869568 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 02:50:06.870739 15852 catalog_manager.cc:4584] TS 40ddd3fa47ea4180b96cc06b4181aa8d (127.28.248.129:35513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40139cd10d384445a70161142afb4fd5: Network error: Client connection negotiation failed: client connection to 127.28.248.129:35513: connect: Connection refused (error 111)
I20260928 02:50:06.875612 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 15834
2026-09-28T02:50:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1167 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:06 dist-test-slave-r238 krb5kdc[16362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:06 dist-test-slave-r238 krb5kdc[16362](info): set up 2 sockets
Sep 28 02:50:06 dist-test-slave-r238 krb5kdc[16362](info): commencing operation
krb5kdc: starting...
W20260928 02:50:07.920792 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:08.047938 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:32909
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:32909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:08.161121 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:08.161453 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:08.161528 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:08.165570 16375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:08.165678 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:08.165717 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:08.165771 16375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:08.165818 16375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:08.171241 16375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:32909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:32909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:08.172417 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:08.173519 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:08.181916 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:08.181919 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:08.181945 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:08.182153 16375 server_base.cc:1034] running on GCE node
I20260928 02:50:08.182993 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:08.184132 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:08.185341 16375 hybrid_clock.cc:648] HybridClock initialized: now 1790563808185329 us; error 30 us; skew 500 ppm
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:08.188218 16375 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:08.189198 16375 webserver.cc:460] Webserver started at http://127.28.248.190:36133/ using document root <none> and password file <none>
I20260928 02:50:08.189718 16375 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:08.189783 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:08.189947 16375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:08.192070 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "13cf45779f284081abd18df5029ee40e"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "f270fe534825c530b783fd465b2d23e0"
server_key_iv: "03fe5103a9581322e546529f7ada4ea3"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.192590 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "13cf45779f284081abd18df5029ee40e"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "f270fe534825c530b783fd465b2d23e0"
server_key_iv: "03fe5103a9581322e546529f7ada4ea3"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.196354 16375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:08.198769 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.199981 16375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:08.200119 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "13cf45779f284081abd18df5029ee40e"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "f270fe534825c530b783fd465b2d23e0"
server_key_iv: "03fe5103a9581322e546529f7ada4ea3"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.200232 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:08.216092 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:08.219655 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:08.219877 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:08.227407 16375 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:32909
I20260928 02:50:08.227417 16442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:32909 every 8 connection(s)
I20260928 02:50:08.228410 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:08.231284 16443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:08.234548 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16375
I20260928 02:50:08.234709 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:08.235000 29666 external_mini_cluster.cc:1428] Setting key d85ad479620fef1a9da9d76c710709ca
I20260928 02:50:08.237402 16443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e: Bootstrap starting.
I20260928 02:50:08.240104 16443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:08.241024 16443 log.cc:824] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:08.243371 16443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e: No bootstrap required, opened a new log
I20260928 02:50:08.247026 16443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf45779f284081abd18df5029ee40e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32909 } }
I20260928 02:50:08.247251 16443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:08.247323 16443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13cf45779f284081abd18df5029ee40e, State: Initialized, Role: FOLLOWER
I20260928 02:50:08.247773 16443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf45779f284081abd18df5029ee40e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32909 } }
I20260928 02:50:08.247893 16443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:08.247982 16443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:08.248132 16443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:08.249169 16443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf45779f284081abd18df5029ee40e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32909 } }
I20260928 02:50:08.249540 16443 leader_election.cc:302] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13cf45779f284081abd18df5029ee40e; no voters:
I20260928 02:50:08.249910 16443 leader_election.cc:288] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:08.250056 16448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:08.250296 16448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [term 1 LEADER]: Becoming Leader. State: Replica: 13cf45779f284081abd18df5029ee40e, State: Running, Role: LEADER
I20260928 02:50:08.250640 16448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf45779f284081abd18df5029ee40e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32909 } }
I20260928 02:50:08.251223 16443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:08.252892 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13cf45779f284081abd18df5029ee40e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf45779f284081abd18df5029ee40e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32909 } } }
I20260928 02:50:08.253014 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:08.253295 16450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13cf45779f284081abd18df5029ee40e. Latest consensus state: current_term: 1 leader_uuid: "13cf45779f284081abd18df5029ee40e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf45779f284081abd18df5029ee40e" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 32909 } } }
I20260928 02:50:08.253422 16450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:08.253703 16457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:08.257150 16457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:08.257210 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.236432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.28.248.190:32909)
0928 02:50:08.236903 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:08.236922 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:08.236964 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:50:08.237769 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:08.237803 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:08.238208 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:08.238577 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:08.239225 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.240795 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.241899 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.242550 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.242711 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.251037 (+ 8326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:08.251045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:08.251062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:08.251098 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:08.253648 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.254497 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.254503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.254510 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.254638 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.254925 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.254930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.254931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.255339 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:08.255566 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:08.255889 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:50:08.256226 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":162,"threads_started":1}
I20260928 02:50:08.262987 16457 catalog_manager.cc:1347] Generated new cluster ID: 96c50225adc74de1814f16f3ac90cc2f
I20260928 02:50:08.263065 16457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:08.272496 16457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:08.273850 16457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:08.283553 16457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e: Generated new TSK 0
I20260928 02:50:08.284438 16457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.248.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
Entry for principal kudu/127.28.248.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab.
I20260928 02:50:08.315759 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:32909
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:08.429626 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:08.429893 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:08.429937 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:08.433990 16469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:08.434051 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:08.434111 16469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:50:08.439462 16469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.129.keytab
--principal=kudu/127.28.248.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:32909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:08.440514 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:08.441721 16469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:08.451373 16477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:08.451424 16474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:08.451427 16475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:08.451609 16469 server_base.cc:1034] running on GCE node
I20260928 02:50:08.452281 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:08.453056 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:08.454258 16469 hybrid_clock.cc:648] HybridClock initialized: now 1790563808454234 us; error 62 us; skew 500 ppm
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:08.457175 16469 init.cc:377] Logged in from keytab as kudu/127.28.248.129@KRBTEST.COM (short username kudu)
I20260928 02:50:08.458000 16469 webserver.cc:460] Webserver started at http://127.28.248.129:36413/ using document root <none> and password file <none>
I20260928 02:50:08.458514 16469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:08.458571 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:08.458743 16469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:08.460834 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "a366829ab5d300c6312eef26b8034954"
server_key_iv: "f0848b6cccf13c93b4207b0c8d893f79"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.461377 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "a366829ab5d300c6312eef26b8034954"
server_key_iv: "f0848b6cccf13c93b4207b0c8d893f79"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.465238 16469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:50:08.467712 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.469215 16469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:08.469348 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "a366829ab5d300c6312eef26b8034954"
server_key_iv: "f0848b6cccf13c93b4207b0c8d893f79"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.469465 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:08.497169 16469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:08.500288 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:08.500509 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:08.501175 16469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:08.502022 16469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:08.502066 16469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.502135 16469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:08.502166 16469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.511435 16469 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:38769
I20260928 02:50:08.511451 16597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:38769 every 8 connection(s)
I20260928 02:50:08.512365 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:08.512941 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16469
I20260928 02:50:08.513075 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:08.513430 29666 external_mini_cluster.cc:1428] Setting key 894ca8b09ff92aec1b04c50c9229637e
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.129@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.130@KRBTEST.COM" created.
I20260928 02:50:08.526865 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.514258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:35141 (local address 127.28.248.190:32909)
0928 02:50:08.514394 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:08.514398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:08.515223 (+ 825us) server_negotiation.cc:407] Connection header received
0928 02:50:08.516558 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:08.516562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:08.516648 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:08.516763 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:08.518057 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.518604 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.519766 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.520007 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.520120 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.522944 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:08.522947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:08.522949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:08.522976 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:08.524344 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.525004 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.525007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.525008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.525090 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.525451 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.525454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.525455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.525715 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:08.525804 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:08.526385 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 02:50:08.526544 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:08.527484 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.514574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:32909 (local address 127.28.248.129:35141)
0928 02:50:08.515053 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:08.515108 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:08.516335 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:08.516949 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:08.516956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:08.517413 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:08.517903 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:08.517917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.518732 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.518736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:08.519635 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:08.519644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.520167 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.520171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:08.520358 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.521016 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:08.521033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:08.522749 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:08.524472 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:08.524478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:08.524490 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:08.524904 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:08.525186 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:08.525188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:08.525190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:08.525330 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:08.525832 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:08.525841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:08.526127 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 02:50:08.526308 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:50:08.526578 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":144,"threads_started":1}
I20260928 02:50:08.528923 16598 heartbeater.cc:347] Connected to a master server at 127.28.248.190:32909
I20260928 02:50:08.529212 16598 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:08.529723 16598 heartbeater.cc:511] Master 127.28.248.190:32909 requested a full tablet report, sending...
I20260928 02:50:08.531288 16407 ts_manager.cc:194] Registered new tserver with Master: 3edc7a9f0baf4c37be8762f8c5ec0e11 (127.28.248.129:38769)
I20260928 02:50:08.532424 16407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.129@KRBTEST.COM'} at 127.28.248.129:35141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
Entry for principal kudu/127.28.248.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab.
I20260928 02:50:08.544396 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:32909
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:08.656900 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:08.657163 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:08.657209 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:08.661374 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:08.661437 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:08.661523 16603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:50:08.667311 16603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.130.keytab
--principal=kudu/127.28.248.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:32909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:08.668359 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:08.669416 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:08.679497 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:08.679481 16608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:08.679543 16603 server_base.cc:1034] running on GCE node
W20260928 02:50:08.679477 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:08.680105 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:08.680796 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:08.681979 16603 hybrid_clock.cc:648] HybridClock initialized: now 1790563808681963 us; error 29 us; skew 500 ppm
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:08.684676 16603 init.cc:377] Logged in from keytab as kudu/127.28.248.130@KRBTEST.COM (short username kudu)
I20260928 02:50:08.685537 16603 webserver.cc:460] Webserver started at http://127.28.248.130:38933/ using document root <none> and password file <none>
I20260928 02:50:08.686095 16603 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:08.686183 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:08.686410 16603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:08.688429 16603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "3eb5cde3048e4bc7974c87ad6a4be72c"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "9c7f1d15eb81e8cf65e1fa4e51b854ff"
server_key_iv: "354136e5e4169bbb8170fa87fa5551d8"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.689036 16603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "3eb5cde3048e4bc7974c87ad6a4be72c"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "9c7f1d15eb81e8cf65e1fa4e51b854ff"
server_key_iv: "354136e5e4169bbb8170fa87fa5551d8"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.692689 16603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:50:08.695024 16618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.696214 16603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:50:08.696364 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "3eb5cde3048e4bc7974c87ad6a4be72c"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "9c7f1d15eb81e8cf65e1fa4e51b854ff"
server_key_iv: "354136e5e4169bbb8170fa87fa5551d8"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.696475 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:08.722944 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:08.726006 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:08.726161 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:08.726680 16603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:08.727460 16603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:08.727505 16603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.727545 16603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:08.727561 16603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.736987 16603 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:37487
I20260928 02:50:08.737056 16731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:37487 every 8 connection(s)
I20260928 02:50:08.737936 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:08.741204 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16603
I20260928 02:50:08.741318 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:08.741623 29666 external_mini_cluster.cc:1428] Setting key b655373fc1abc2e54fcbd0647b927ed5
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.130@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.131@KRBTEST.COM; defaulting to no policy
I20260928 02:50:08.753862 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.739757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:49641 (local address 127.28.248.190:32909)
0928 02:50:08.739908 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:08.739913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:08.740858 (+ 945us) server_negotiation.cc:407] Connection header received
0928 02:50:08.742425 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:08.742428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:08.742467 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:08.742538 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:08.743883 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.744406 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.745685 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.746110 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.746233 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.749474 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:08.749477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:08.749479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:08.749504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:08.751053 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.751944 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.751948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.751949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.752046 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.752519 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.752522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.752524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.752818 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:08.752912 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:08.753597 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 02:50:08.753728 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.131@KRBTEST.COM" created.
I20260928 02:50:08.754931 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.740216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:32909 (local address 127.28.248.130:49641)
0928 02:50:08.740717 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:08.740760 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:08.742187 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:08.742731 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:08.742739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:08.743232 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:08.743733 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:08.743747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.744576 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.744583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:08.745558 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:08.745567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.746250 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.746253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:08.746405 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.747144 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:08.747161 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:08.749278 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:08.751186 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:08.751195 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:08.751215 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:08.751735 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:08.752163 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:08.752166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:08.752168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:08.752389 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:08.752929 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:08.752938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:08.753302 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:50:08.753516 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:50:08.753801 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260928 02:50:08.756215 16732 heartbeater.cc:347] Connected to a master server at 127.28.248.190:32909
I20260928 02:50:08.756489 16732 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:08.757166 16732 heartbeater.cc:511] Master 127.28.248.190:32909 requested a full tablet report, sending...
I20260928 02:50:08.758437 16407 ts_manager.cc:194] Registered new tserver with Master: 3eb5cde3048e4bc7974c87ad6a4be72c (127.28.248.130:37487)
I20260928 02:50:08.759297 16407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.130@KRBTEST.COM'} at 127.28.248.130:49641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
Entry for principal kudu/127.28.248.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab.
I20260928 02:50:08.771579 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:32909
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:08.883508 16737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:08.883783 16737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:08.883826 16737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:08.887765 16737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:08.887822 16737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:08.887881 16737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:50:08.893147 16737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.131.keytab
--principal=kudu/127.28.248.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:32909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:08.894176 16737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:08.895151 16737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:08.905230 16743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:08.905237 16742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:08.905303 16737 server_base.cc:1034] running on GCE node
W20260928 02:50:08.905246 16745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:08.905867 16737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:08.906517 16737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:08.907672 16737 hybrid_clock.cc:648] HybridClock initialized: now 1790563808907653 us; error 31 us; skew 500 ppm
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:08.910432 16737 init.cc:377] Logged in from keytab as kudu/127.28.248.131@KRBTEST.COM (short username kudu)
I20260928 02:50:08.911309 16737 webserver.cc:460] Webserver started at http://127.28.248.131:40913/ using document root <none> and password file <none>
I20260928 02:50:08.911914 16737 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:08.912009 16737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:08.912242 16737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:08.914392 16737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "ce354e5ccc384311b6f53a564524cf52"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "f6ec75f538cacf9eb335522ebc1d4a60"
server_key_iv: "65e30be26942c9a92161f821669dcf29"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.914955 16737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "ce354e5ccc384311b6f53a564524cf52"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "f6ec75f538cacf9eb335522ebc1d4a60"
server_key_iv: "65e30be26942c9a92161f821669dcf29"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.919006 16737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:50:08.921641 16752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.922973 16737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:08.923122 16737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "ce354e5ccc384311b6f53a564524cf52"
format_stamp: "Formatted at 2026-09-28 02:50:08 on dist-test-slave-r238"
server_key: "f6ec75f538cacf9eb335522ebc1d4a60"
server_key_iv: "65e30be26942c9a92161f821669dcf29"
server_key_version: "encryptionkey@0"
I20260928 02:50:08.923249 16737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:08.944550 16737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:08.947844 16737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:08.948062 16737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:08.948743 16737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:08.949601 16737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:08.949676 16737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.949745 16737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:08.949792 16737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:08.959168 16737 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:32775
I20260928 02:50:08.959188 16865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:32775 every 8 connection(s)
I20260928 02:50:08.960093 16737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:08.968434 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16737
I20260928 02:50:08.968595 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:08.968953 29666 external_mini_cluster.cc:1428] Setting key dcc65fdf12e0e5b4991f78049637604a
Sep 28 02:50:08 dist-test-slave-r238 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563808, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.131@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:08.974272 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.961953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:48189 (local address 127.28.248.190:32909)
0928 02:50:08.962096 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:08.962100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:08.962921 (+ 821us) server_negotiation.cc:407] Connection header received
0928 02:50:08.964189 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:08.964192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:08.964243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:08.964320 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:08.965549 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.966120 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.967296 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.967540 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.967658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.970601 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:08.970604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:08.970606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:08.970631 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:08.972123 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.972798 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.972801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.972802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.972880 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.973217 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.973219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.973220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.973453 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:08.973532 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:08.974046 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 02:50:08.974151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:08.975168 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.962294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:32909 (local address 127.28.248.131:48189)
0928 02:50:08.962728 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:08.962777 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:08.963977 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:08.964454 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:08.964461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:08.964910 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:08.965399 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:08.965414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.966241 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.966243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:08.967165 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:08.967173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.967701 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.967705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:08.967879 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.968441 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:08.968465 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:08.970454 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:08.972262 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:08.972267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:08.972280 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:08.972702 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:08.972974 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:08.972976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:08.972978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:08.973121 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:08.973555 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:08.973560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:08.973825 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:50:08.973982 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:50:08.974276 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260928 02:50:08.976336 16866 heartbeater.cc:347] Connected to a master server at 127.28.248.190:32909
I20260928 02:50:08.976560 16866 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:08.977085 16866 heartbeater.cc:511] Master 127.28.248.190:32909 requested a full tablet report, sending...
I20260928 02:50:08.978139 16407 ts_manager.cc:194] Registered new tserver with Master: ce354e5ccc384311b6f53a564524cf52 (127.28.248.131:32775)
I20260928 02:50:08.978685 16407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.248.131@KRBTEST.COM'} at 127.28.248.131:48189
I20260928 02:50:08.983263 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:08.992784 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:08.985318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.28.248.190:32909)
0928 02:50:08.985451 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:08.985455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:08.985609 (+ 154us) server_negotiation.cc:407] Connection header received
0928 02:50:08.985728 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:08.985730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:08.985770 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:08.985841 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:08.986507 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.987031 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.987889 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:08.988099 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:08.988173 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:08.989615 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:08.989618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:08.989620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:08.989644 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:08.991099 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.991576 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.991579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.991580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.991655 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:08.991932 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:08.991934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:08.991935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:08.992151 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:08.992253 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:08.992469 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:50:08.992567 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:50:08.995642 16407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:08.998922 16407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:09.014111 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.009150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.28.248.129:38769)
0928 02:50:09.009538 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.009583 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.009605 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:50:09.009674 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.009679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.009875 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.010023 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:09.010202 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.010920 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.012677 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.013380 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.013525 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.013666 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.013824 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.013931 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":92,"threads_started":1}
I20260928 02:50:09.014093 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.008207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:38769 (local address 127.0.0.1:49954)
0928 02:50:09.008969 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.008980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:09.009062 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:09.010007 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:09.010010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:09.010027 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:09.010108 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.010114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.011051 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.011057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.012503 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.012515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.013510 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.013513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.013628 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.013665 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:50:09.013795 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:50:09.013922 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260928 02:50:09.014554 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.008008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.28.248.130:37487)
0928 02:50:09.009293 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.009304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.009324 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:50:09.009389 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.009392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.009535 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.009654 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:09.010086 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.011189 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.012867 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.013646 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.013760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.013911 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.014311 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.014404 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":143,"threads_started":1}
I20260928 02:50:09.014761 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.008737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.28.248.131:32775)
0928 02:50:09.009111 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.009125 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.009153 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:50:09.009231 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.009236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.009466 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.009629 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:09.009935 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.011184 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.012702 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.013583 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.013868 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.013977 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.014523 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.014623 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":144,"threads_started":1}
I20260928 02:50:09.014428 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.007958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:32775 (local address 127.0.0.1:59296)
0928 02:50:09.008721 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.008733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:09.008801 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:09.009642 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:09.009648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:09.009672 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:09.009809 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.009815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.011363 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.011366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.012583 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.012594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.014018 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.014022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.014123 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.014145 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:50:09.014205 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:50:09.014279 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":88,"thread_start_us":112,"threads_started":1}
I20260928 02:50:09.014982 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.007880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:37487 (local address 127.0.0.1:51062)
0928 02:50:09.008524 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.008574 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:09.008731 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:09.009819 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:09.009822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:09.009838 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:09.009955 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.009961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.011531 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.011535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.012677 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.012687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.013973 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.013977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.014072 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.014115 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:50:09.014190 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:50:09.014702 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":505,"thread_start_us":90,"threads_started":1}
I20260928 02:50:09.017707 16666 tablet_service.cc:1467] Processing CreateTablet for tablet abdf50a6db824ea2ba4694c88c916467 (DEFAULT_TABLE table=test-table [id=1686e727d1e84e2bb2494e42590e9194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:09.017705 16800 tablet_service.cc:1467] Processing CreateTablet for tablet abdf50a6db824ea2ba4694c88c916467 (DEFAULT_TABLE table=test-table [id=1686e727d1e84e2bb2494e42590e9194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:09.017714 16532 tablet_service.cc:1467] Processing CreateTablet for tablet abdf50a6db824ea2ba4694c88c916467 (DEFAULT_TABLE table=test-table [id=1686e727d1e84e2bb2494e42590e9194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:09.018786 16532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdf50a6db824ea2ba4694c88c916467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:09.018793 16800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdf50a6db824ea2ba4694c88c916467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:09.018793 16666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdf50a6db824ea2ba4694c88c916467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:09.027872 16884 tablet_bootstrap.cc:492] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Bootstrap starting.
I20260928 02:50:09.027977 16883 tablet_bootstrap.cc:492] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Bootstrap starting.
I20260928 02:50:09.028949 16885 tablet_bootstrap.cc:492] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: Bootstrap starting.
I20260928 02:50:09.030483 16884 tablet_bootstrap.cc:654] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:09.031169 16885 tablet_bootstrap.cc:654] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:09.031865 16884 log.cc:824] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:09.031921 16885 log.cc:824] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:09.033953 16883 tablet_bootstrap.cc:654] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:09.034233 16884 tablet_bootstrap.cc:492] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: No bootstrap required, opened a new log
I20260928 02:50:09.034474 16884 ts_tablet_manager.cc:1397] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:50:09.035220 16883 log.cc:824] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:09.037364 16883 tablet_bootstrap.cc:492] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: No bootstrap required, opened a new log
I20260928 02:50:09.037586 16883 ts_tablet_manager.cc:1397] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260928 02:50:09.038044 16885 tablet_bootstrap.cc:492] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: No bootstrap required, opened a new log
I20260928 02:50:09.038232 16885 ts_tablet_manager.cc:1397] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 02:50:09.038581 16884 raft_consensus.cc:348] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.038878 16884 raft_consensus.cc:374] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:09.038949 16884 raft_consensus.cc:729] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb5cde3048e4bc7974c87ad6a4be72c, State: Initialized, Role: FOLLOWER
I20260928 02:50:09.039429 16884 consensus_queue.cc:260] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.040162 16732 heartbeater.cc:503] Master 127.28.248.190:32909 was elected leader, sending a full tablet report...
I20260928 02:50:09.040567 16884 ts_tablet_manager.cc:1428] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:50:09.041254 16883 raft_consensus.cc:348] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.041513 16883 raft_consensus.cc:374] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:09.041574 16883 raft_consensus.cc:729] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce354e5ccc384311b6f53a564524cf52, State: Initialized, Role: FOLLOWER
I20260928 02:50:09.042004 16883 consensus_queue.cc:260] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.042682 16866 heartbeater.cc:503] Master 127.28.248.190:32909 was elected leader, sending a full tablet report...
I20260928 02:50:09.042600 16885 raft_consensus.cc:348] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.042841 16885 raft_consensus.cc:374] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:09.042891 16885 raft_consensus.cc:729] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edc7a9f0baf4c37be8762f8c5ec0e11, State: Initialized, Role: FOLLOWER
I20260928 02:50:09.042994 16883 ts_tablet_manager.cc:1428] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260928 02:50:09.043323 16885 consensus_queue.cc:260] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.044320 16885 ts_tablet_manager.cc:1428] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 02:50:09.044737 16598 heartbeater.cc:503] Master 127.28.248.190:32909 was elected leader, sending a full tablet report...
W20260928 02:50:09.211086 16867 tablet.cc:2369] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:09.239068 16733 tablet.cc:2369] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:09.264051 16599 tablet.cc:2369] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:09.269785 16890 raft_consensus.cc:482] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:09.270011 16890 raft_consensus.cc:504] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.271239 16890 leader_election.cc:288] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb5cde3048e4bc7974c87ad6a4be72c (127.28.248.130:37487), 3edc7a9f0baf4c37be8762f8c5ec0e11 (127.28.248.129:38769)
I20260928 02:50:09.275585 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.271505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:38769 (local address 127.28.248.131:49401)
0928 02:50:09.271621 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.271637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:09.271738 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:09.272047 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:09.272050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:09.272067 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:09.272187 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.272193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.273217 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.273220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.274407 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.274416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.275120 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.275124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.275232 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.275267 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:50:09.275362 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 02:50:09.275440 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":32}
I20260928 02:50:09.275616 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.271618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:49401 (local address 127.28.248.129:38769)
0928 02:50:09.271752 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.271756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.271776 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:50:09.271890 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.271892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.271937 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.272005 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:09.272319 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.273076 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.274529 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.275002 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.275096 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.275138 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.275389 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.275469 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:50:09.275767 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.271520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:37487 (local address 127.28.248.131:46417)
0928 02:50:09.272199 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.272210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:09.272325 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:09.272540 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:09.272543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:09.272561 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:09.272709 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.272715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.273694 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.273696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.274604 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.274615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.275372 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.275376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.275467 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.275490 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:50:09.275564 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:50:09.275622 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":345,"thread_start_us":132,"threads_started":1}
I20260928 02:50:09.275810 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.271581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:46417 (local address 127.28.248.130:37487)
0928 02:50:09.271707 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.271712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.272324 (+ 612us) server_negotiation.cc:407] Connection header received
0928 02:50:09.272408 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.272411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.272452 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.272511 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:09.272893 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.273584 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.274733 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.275230 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.275354 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.275402 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.275593 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.275679 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:09.276211 16552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdf50a6db824ea2ba4694c88c916467" candidate_uuid: "ce354e5ccc384311b6f53a564524cf52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" is_pre_election: true
I20260928 02:50:09.276481 16686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdf50a6db824ea2ba4694c88c916467" candidate_uuid: "ce354e5ccc384311b6f53a564524cf52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" is_pre_election: true
I20260928 02:50:09.276573 16552 raft_consensus.cc:2454] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce354e5ccc384311b6f53a564524cf52 in term 0.
I20260928 02:50:09.276769 16686 raft_consensus.cc:2454] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce354e5ccc384311b6f53a564524cf52 in term 0.
I20260928 02:50:09.277084 16754 leader_election.cc:302] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3edc7a9f0baf4c37be8762f8c5ec0e11, ce354e5ccc384311b6f53a564524cf52; no voters:
I20260928 02:50:09.277400 16890 raft_consensus.cc:2781] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:09.277478 16890 raft_consensus.cc:482] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:09.277523 16890 raft_consensus.cc:3037] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:09.278548 16890 raft_consensus.cc:504] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.278906 16890 leader_election.cc:288] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb5cde3048e4bc7974c87ad6a4be72c (127.28.248.130:37487), 3edc7a9f0baf4c37be8762f8c5ec0e11 (127.28.248.129:38769)
I20260928 02:50:09.279201 16686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdf50a6db824ea2ba4694c88c916467" candidate_uuid: "ce354e5ccc384311b6f53a564524cf52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c"
I20260928 02:50:09.279300 16552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdf50a6db824ea2ba4694c88c916467" candidate_uuid: "ce354e5ccc384311b6f53a564524cf52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11"
I20260928 02:50:09.279330 16686 raft_consensus.cc:3037] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:09.279446 16552 raft_consensus.cc:3037] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:09.280359 16686 raft_consensus.cc:2454] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce354e5ccc384311b6f53a564524cf52 in term 1.
I20260928 02:50:09.280495 16552 raft_consensus.cc:2454] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce354e5ccc384311b6f53a564524cf52 in term 1.
I20260928 02:50:09.280670 16753 leader_election.cc:302] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb5cde3048e4bc7974c87ad6a4be72c, ce354e5ccc384311b6f53a564524cf52; no voters:
I20260928 02:50:09.280843 16890 raft_consensus.cc:2781] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:09.281221 16890 raft_consensus.cc:686] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 1 LEADER]: Becoming Leader. State: Replica: ce354e5ccc384311b6f53a564524cf52, State: Running, Role: LEADER
I20260928 02:50:09.281581 16890 consensus_queue.cc:237] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } }
I20260928 02:50:09.284822 16406 catalog_manager.cc:5495] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce354e5ccc384311b6f53a564524cf52 (127.28.248.131). New cstate: current_term: 1 leader_uuid: "ce354e5ccc384311b6f53a564524cf52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce354e5ccc384311b6f53a564524cf52" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 32775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 } health_report { overall_health: UNKNOWN } } }
I20260928 02:50:09.313191 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.309886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.28.248.131:32775)
0928 02:50:09.310013 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.310017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.310030 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:09.310103 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.310105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.310158 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.310245 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:09.310462 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.310952 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.311760 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.311969 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.312041 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.312291 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:09.312784 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:09.312878 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.312943 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.312994 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:50:09.319005 16686 raft_consensus.cc:1261] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 1 FOLLOWER]: Refusing update from remote peer ce354e5ccc384311b6f53a564524cf52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:09.319079 16552 raft_consensus.cc:1261] T abdf50a6db824ea2ba4694c88c916467 P 3edc7a9f0baf4c37be8762f8c5ec0e11 [term 1 FOLLOWER]: Refusing update from remote peer ce354e5ccc384311b6f53a564524cf52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:09.319756 16893 consensus_queue.cc:1035] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3edc7a9f0baf4c37be8762f8c5ec0e11" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:09.319927 16890 consensus_queue.cc:1035] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb5cde3048e4bc7974c87ad6a4be72c" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:09.326121 16896 mvcc.cc:205] Tried to move back new op lower bound from 7334149362963357696 to 7334149362820096000. Current Snapshot: MvccSnapshot[applied={T|T < 7334149362963357696}]
I20260928 02:50:09.327332 16898 mvcc.cc:205] Tried to move back new op lower bound from 7334149362963357696 to 7334149362820096000. Current Snapshot: MvccSnapshot[applied={T|T < 7334149362963357696}]
I20260928 02:50:09.328018 16897 mvcc.cc:205] Tried to move back new op lower bound from 7334149362963357696 to 7334149362820096000. Current Snapshot: MvccSnapshot[applied={T|T < 7334149362963357696}]
I20260928 02:50:09.335136 16407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:09.335273 16407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:09.337430 16407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13cf45779f284081abd18df5029ee40e: Sending DeleteTablet for 3 replicas of tablet abdf50a6db824ea2ba4694c88c916467
I20260928 02:50:09.338132 16800 tablet_service.cc:1514] Processing DeleteTablet for tablet abdf50a6db824ea2ba4694c88c916467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:09 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:59296
I20260928 02:50:09.338155 16666 tablet_service.cc:1514] Processing DeleteTablet for tablet abdf50a6db824ea2ba4694c88c916467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:09 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:51062
I20260928 02:50:09.338446 16532 tablet_service.cc:1514] Processing DeleteTablet for tablet abdf50a6db824ea2ba4694c88c916467 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:09 UTC) from {username='kudu', principal='kudu/127.28.248.190@KRBTEST.COM'} at 127.0.0.1:49954
I20260928 02:50:09.338626 16907 tablet_replica.cc:330] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: stopping tablet replica
I20260928 02:50:09.338891 16906 tablet_replica.cc:330] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: stopping tablet replica
I20260928 02:50:09.338960 16907 raft_consensus.cc:2229] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:09.339123 16906 raft_consensus.cc:2229] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:09.339253 16907 raft_consensus.cc:2258] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:09.339462 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16469
I20260928 02:50:09.339460 16906 raft_consensus.cc:2258] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:09.341814 16906 ts_tablet_manager.cc:1905] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:09.342268 16907 ts_tablet_manager.cc:1905] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:09.344578 16906 ts_tablet_manager.cc:1918] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:09.344702 16906 log.cc:1196] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/abdf50a6db824ea2ba4694c88c916467
I20260928 02:50:09.344882 16907 ts_tablet_manager.cc:1918] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:09.344980 16907 log.cc:1196] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/abdf50a6db824ea2ba4694c88c916467
I20260928 02:50:09.345031 16906 ts_tablet_manager.cc:1939] T abdf50a6db824ea2ba4694c88c916467 P ce354e5ccc384311b6f53a564524cf52: Deleting consensus metadata
I20260928 02:50:09.345283 16907 ts_tablet_manager.cc:1939] T abdf50a6db824ea2ba4694c88c916467 P 3eb5cde3048e4bc7974c87ad6a4be72c: Deleting consensus metadata
I20260928 02:50:09.346216 16391 catalog_manager.cc:4841] TS 3eb5cde3048e4bc7974c87ad6a4be72c (127.28.248.130:37487): tablet abdf50a6db824ea2ba4694c88c916467 (table test-table [id=1686e727d1e84e2bb2494e42590e9194]) successfully deleted
I20260928 02:50:09.346189 16394 catalog_manager.cc:4841] TS ce354e5ccc384311b6f53a564524cf52 (127.28.248.131:32775): tablet abdf50a6db824ea2ba4694c88c916467 (table test-table [id=1686e727d1e84e2bb2494e42590e9194]) successfully deleted
W20260928 02:50:09.348170 16392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:38769 (error 108)
I20260928 02:50:09.348388 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16603
I20260928 02:50:09.350732 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.350431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:38769 (local address 127.0.0.1:49970)
0928 02:50:09.350575 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.350647 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:50:09.350907 16392 catalog_manager.cc:4584] TS 3edc7a9f0baf4c37be8762f8c5ec0e11 (127.28.248.129:38769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abdf50a6db824ea2ba4694c88c916467: Network error: Client connection negotiation failed: client connection to 127.28.248.129:38769: connect: Connection refused (error 111)
I20260928 02:50:09.354539 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16737
I20260928 02:50:09.359270 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16375
2026-09-28T02:50:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2483 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:09Z Disabled control of system clock
I20260928 02:50:09.388551 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36251
--webserver_interface=127.28.248.190
--webserver_port=0
--builtin_ntp_servers=127.28.248.148:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:50:09.496147 16913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:09.496400 16913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:09.496435 16913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:09.500392 16913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:09.500450 16913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:09.500465 16913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:09.500481 16913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:09.500496 16913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:09.505178 16913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:36251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:09.506152 16913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:09.507468 16913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:09.515718 16913 server_base.cc:1034] running on GCE node
W20260928 02:50:09.515689 16919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:09.515663 16921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:09.515667 16918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:09.516446 16913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:09.517607 16913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:09.518791 16913 hybrid_clock.cc:648] HybridClock initialized: now 1790563809518778 us; error 37 us; skew 500 ppm
I20260928 02:50:09.520743 16913 webserver.cc:460] Webserver started at http://127.28.248.190:37267/ using document root <none> and password file <none>
I20260928 02:50:09.521308 16913 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:09.521405 16913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:09.521669 16913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:09.523895 16913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "700dc00ac7074b449edad57aa18509fc"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "3708879721c8f44191ccdbdba4666991"
server_key_iv: "7b618141ec5af657f406e6007c0fe609"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.524463 16913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "700dc00ac7074b449edad57aa18509fc"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "3708879721c8f44191ccdbdba4666991"
server_key_iv: "7b618141ec5af657f406e6007c0fe609"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.528071 16913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:50:09.530395 16927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:09.531562 16913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:09.531692 16913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "700dc00ac7074b449edad57aa18509fc"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "3708879721c8f44191ccdbdba4666991"
server_key_iv: "7b618141ec5af657f406e6007c0fe609"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.531793 16913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:09.561964 16913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:09.562682 16913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:09.562882 16913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:09.569502 16913 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:36251
I20260928 02:50:09.569512 16979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:36251 every 8 connection(s)
I20260928 02:50:09.570436 16913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:09.573498 16980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:09.573854 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 16913
I20260928 02:50:09.573979 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:09.574290 29666 external_mini_cluster.cc:1428] Setting key 1d22adbd0be2de6bbbe6f1f18e4c43bb
I20260928 02:50:09.579676 16980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc: Bootstrap starting.
I20260928 02:50:09.582294 16980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:09.583117 16980 log.cc:824] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:09.583478 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.575600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.28.248.190:36251)
0928 02:50:09.576184 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.576199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.576235 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:50:09.576951 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.576970 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.577377 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.577659 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:09.578041 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.578979 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.580141 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.580635 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.580759 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.581752 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:09.581757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:09.581769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:09.582147 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:09.582203 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:09.582221 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:09.582327 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.582456 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.582704 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":132,"threads_started":1}
I20260928 02:50:09.585028 16980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc: No bootstrap required, opened a new log
I20260928 02:50:09.588380 16980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700dc00ac7074b449edad57aa18509fc" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36251 } }
I20260928 02:50:09.588594 16980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:09.588718 16980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700dc00ac7074b449edad57aa18509fc, State: Initialized, Role: FOLLOWER
I20260928 02:50:09.589103 16980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700dc00ac7074b449edad57aa18509fc" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36251 } }
I20260928 02:50:09.589219 16980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:09.589264 16980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:09.589336 16980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:09.590229 16980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700dc00ac7074b449edad57aa18509fc" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36251 } }
I20260928 02:50:09.590514 16980 leader_election.cc:302] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700dc00ac7074b449edad57aa18509fc; no voters:
I20260928 02:50:09.590754 16980 leader_election.cc:288] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:09.590878 16985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:09.591140 16985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [term 1 LEADER]: Becoming Leader. State: Replica: 700dc00ac7074b449edad57aa18509fc, State: Running, Role: LEADER
I20260928 02:50:09.591439 16985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700dc00ac7074b449edad57aa18509fc" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36251 } }
I20260928 02:50:09.592015 16980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:09.593127 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700dc00ac7074b449edad57aa18509fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700dc00ac7074b449edad57aa18509fc" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36251 } } }
I20260928 02:50:09.593253 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:09.593523 16987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700dc00ac7074b449edad57aa18509fc. Latest consensus state: current_term: 1 leader_uuid: "700dc00ac7074b449edad57aa18509fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700dc00ac7074b449edad57aa18509fc" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 36251 } } }
I20260928 02:50:09.593638 16987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:09.593711 16994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:09.596310 16994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:09.601317 16994 catalog_manager.cc:1347] Generated new cluster ID: c599cc8173304bdda023b90ebd4c668f
I20260928 02:50:09.601389 16994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:09.617391 16994 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:09.618745 16994 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:09.631990 16994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc: Generated new TSK 0
I20260928 02:50:09.632946 16994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:09.643004 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.129:0
--local_ip_for_outbound_sockets=127.28.248.129
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36251
--builtin_ntp_servers=127.28.248.148:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:50:09.757876 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:09.758169 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:09.758237 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:09.762323 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:09.762408 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:09.762497 17004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.129
I20260928 02:50:09.767292 17004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.248.129
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.129
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:09.768438 17004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:09.769913 17004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:09.779621 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:09.779609 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:09.779711 17004 server_base.cc:1034] running on GCE node
W20260928 02:50:09.779604 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:09.780282 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:09.781081 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:09.782261 17004 hybrid_clock.cc:648] HybridClock initialized: now 1790563809782244 us; error 38 us; skew 500 ppm
I20260928 02:50:09.784152 17004 webserver.cc:460] Webserver started at http://127.28.248.129:42165/ using document root <none> and password file <none>
I20260928 02:50:09.784797 17004 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:09.784885 17004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:09.785110 17004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:09.787168 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data/instance:
uuid: "a52354de5acf4b69a249806fe294fa89"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "d783262531dd63459036f86daf4b0610"
server_key_iv: "29efb4f6cb582471de4b953ea5cfd4f0"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.787722 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance:
uuid: "a52354de5acf4b69a249806fe294fa89"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "d783262531dd63459036f86daf4b0610"
server_key_iv: "29efb4f6cb582471de4b953ea5cfd4f0"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.791502 17004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:50:09.794175 17018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:09.795470 17004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:09.795606 17004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal
uuid: "a52354de5acf4b69a249806fe294fa89"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "d783262531dd63459036f86daf4b0610"
server_key_iv: "29efb4f6cb582471de4b953ea5cfd4f0"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.795727 17004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:09.824280 17004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:09.825042 17004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:09.825249 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:09.825850 17004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:09.826783 17004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:09.826867 17004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:09.826954 17004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:09.826999 17004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:09.836146 17004 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.129:35623
I20260928 02:50:09.836167 17131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.129:35623 every 8 connection(s)
I20260928 02:50:09.837108 17004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/data/info.pb
I20260928 02:50:09.840034 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17004
I20260928 02:50:09.840160 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-0/wal/instance
I20260928 02:50:09.840447 29666 external_mini_cluster.cc:1428] Setting key fda90c0f1bf7496fba1cd24785612c3a
I20260928 02:50:09.842480 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.130:0
--local_ip_for_outbound_sockets=127.28.248.130
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36251
--builtin_ntp_servers=127.28.248.148:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:50:09.847244 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.838915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:37271 (local address 127.28.248.190:36251)
0928 02:50:09.839057 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:09.839062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:09.840226 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 02:50:09.842104 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:09.842108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:09.842159 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:09.842251 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:09.843205 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.844008 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.845266 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.845507 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.845615 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.846733 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:09.846736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:09.846739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:09.846756 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:09.846774 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:09.846781 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:09.846850 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:09.847040 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:50:09.847103 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:09.848099 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:09.839370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36251 (local address 127.28.248.129:37271)
0928 02:50:09.840034 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:09.840077 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:09.841868 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:09.842411 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:09.842417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:09.842502 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:09.843063 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.843077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.844133 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.844137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.845140 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:09.845149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:09.845650 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:09.845655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:09.845823 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:09.846486 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:09.846488 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:09.846526 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:09.846589 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:09.846903 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:09.846910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:09.846914 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:50:09.847034 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:50:09.847316 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20260928 02:50:09.849342 17132 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36251
I20260928 02:50:09.849607 17132 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:09.850275 17132 heartbeater.cc:511] Master 127.28.248.190:36251 requested a full tablet report, sending...
I20260928 02:50:09.852061 16944 ts_manager.cc:194] Registered new tserver with Master: a52354de5acf4b69a249806fe294fa89 (127.28.248.129:35623)
I20260928 02:50:09.853739 16944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.129:37271
W20260928 02:50:09.953210 17135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:09.953444 17135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:09.953480 17135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:09.957420 17135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:09.957468 17135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:09.957518 17135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.130
I20260928 02:50:09.962123 17135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.248.130
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.130
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:09.963017 17135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:09.964259 17135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:09.973362 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:09.973362 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:09.973413 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:09.973462 17135 server_base.cc:1034] running on GCE node
I20260928 02:50:09.974103 17135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:09.974709 17135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:09.975879 17135 hybrid_clock.cc:648] HybridClock initialized: now 1790563809975860 us; error 28 us; skew 500 ppm
I20260928 02:50:09.977721 17135 webserver.cc:460] Webserver started at http://127.28.248.130:41599/ using document root <none> and password file <none>
I20260928 02:50:09.978304 17135 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:09.978365 17135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:09.978598 17135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:09.980541 17135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data/instance:
uuid: "c31398fc0bd248f39a6f3322740c974a"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "99b3946717904daf65b12e1ca3640424"
server_key_iv: "9e02256642788c8eebcfa6524d449ce3"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.981145 17135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance:
uuid: "c31398fc0bd248f39a6f3322740c974a"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "99b3946717904daf65b12e1ca3640424"
server_key_iv: "9e02256642788c8eebcfa6524d449ce3"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.984658 17135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:50:09.992468 17149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:09.994244 17135 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260928 02:50:09.994377 17135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal
uuid: "c31398fc0bd248f39a6f3322740c974a"
format_stamp: "Formatted at 2026-09-28 02:50:09 on dist-test-slave-r238"
server_key: "99b3946717904daf65b12e1ca3640424"
server_key_iv: "9e02256642788c8eebcfa6524d449ce3"
server_key_version: "encryptionkey@0"
I20260928 02:50:09.994499 17135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:10.005055 17135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:10.005717 17135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:10.005918 17135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:10.006505 17135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:10.007428 17135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:10.007474 17135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:10.007541 17135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:10.007579 17135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:10.016059 17135 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.130:38977
I20260928 02:50:10.016080 17262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.130:38977 every 8 connection(s)
I20260928 02:50:10.017004 17135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/data/info.pb
I20260928 02:50:10.021955 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17135
I20260928 02:50:10.022094 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal/instance
I20260928 02:50:10.022409 29666 external_mini_cluster.cc:1428] Setting key b399be4d3dba67854f9b0436894e2e0e
I20260928 02:50:10.024271 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.248.131:0
--local_ip_for_outbound_sockets=127.28.248.131
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36251
--builtin_ntp_servers=127.28.248.148:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:50:10.026862 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.018800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.130:46685 (local address 127.28.248.190:36251)
0928 02:50:10.018954 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.018959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.020235 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 02:50:10.021783 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.021787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.021855 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.021960 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:10.022957 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.023674 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.024969 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.025203 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.025316 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.026372 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:10.026376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:10.026378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:10.026391 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:10.026413 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:10.026428 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:10.026497 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.026659 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.026721 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:50:10.027710 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.019297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36251 (local address 127.28.248.130:46685)
0928 02:50:10.020084 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.020125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.021577 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.022153 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.022166 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.022270 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:10.022815 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.022828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.023796 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.023799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.024847 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.024857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.025344 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.025348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.025512 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.026129 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:10.026131 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:10.026170 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:10.026231 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:10.026511 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:10.026526 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:10.026532 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.026645 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.026962 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":144,"threads_started":1}
I20260928 02:50:10.028928 17263 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36251
I20260928 02:50:10.029202 17263 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:10.029888 17263 heartbeater.cc:511] Master 127.28.248.190:36251 requested a full tablet report, sending...
I20260928 02:50:10.031109 16944 ts_manager.cc:194] Registered new tserver with Master: c31398fc0bd248f39a6f3322740c974a (127.28.248.130:38977)
I20260928 02:50:10.032015 16944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.130:46685
W20260928 02:50:10.136711 17266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:10.136965 17266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:10.137001 17266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:10.140930 17266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:10.140977 17266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:10.141031 17266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.248.131
I20260928 02:50:10.145576 17266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.248.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.248.131
--webserver_port=0
--tserver_master_addrs=127.28.248.190:36251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.248.131
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:10.146548 17266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:10.147814 17266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:10.156978 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:10.157102 17266 server_base.cc:1034] running on GCE node
W20260928 02:50:10.157169 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:10.156947 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:10.157572 17266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:10.158140 17266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:10.159322 17266 hybrid_clock.cc:648] HybridClock initialized: now 1790563810159311 us; error 27 us; skew 500 ppm
I20260928 02:50:10.161150 17266 webserver.cc:460] Webserver started at http://127.28.248.131:39961/ using document root <none> and password file <none>
I20260928 02:50:10.161712 17266 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:10.161801 17266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:10.162029 17266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:10.164063 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data/instance:
uuid: "2026c7946c1641c7b4159248fb69f65d"
format_stamp: "Formatted at 2026-09-28 02:50:10 on dist-test-slave-r238"
server_key: "69d10b1d5202d69e62572c3a046cd53f"
server_key_iv: "f8799b3af4d6c5b96b12608bd05cb4a9"
server_key_version: "encryptionkey@0"
I20260928 02:50:10.164657 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance:
uuid: "2026c7946c1641c7b4159248fb69f65d"
format_stamp: "Formatted at 2026-09-28 02:50:10 on dist-test-slave-r238"
server_key: "69d10b1d5202d69e62572c3a046cd53f"
server_key_iv: "f8799b3af4d6c5b96b12608bd05cb4a9"
server_key_version: "encryptionkey@0"
I20260928 02:50:10.168274 17266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:50:10.170889 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:10.172166 17266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:10.172302 17266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal
uuid: "2026c7946c1641c7b4159248fb69f65d"
format_stamp: "Formatted at 2026-09-28 02:50:10 on dist-test-slave-r238"
server_key: "69d10b1d5202d69e62572c3a046cd53f"
server_key_iv: "f8799b3af4d6c5b96b12608bd05cb4a9"
server_key_version: "encryptionkey@0"
I20260928 02:50:10.172410 17266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:10.189006 17266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:10.189738 17266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:10.189949 17266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:10.190554 17266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:50:10.191550 17266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:50:10.191619 17266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:10.191685 17266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:50:10.191730 17266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:10.200682 17266 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.131:41109
I20260928 02:50:10.200814 17393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.131:41109 every 8 connection(s)
I20260928 02:50:10.201623 17266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/data/info.pb
I20260928 02:50:10.210788 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17266
I20260928 02:50:10.210917 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal/instance
I20260928 02:50:10.211226 29666 external_mini_cluster.cc:1428] Setting key 43fb21377828fcb4487d06102e46ff15
I20260928 02:50:10.211763 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.203599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.131:36315 (local address 127.28.248.190:36251)
0928 02:50:10.203752 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.203757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.204977 (+ 1220us) server_negotiation.cc:407] Connection header received
0928 02:50:10.206413 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.206416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.206463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.206538 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:10.207480 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.207999 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.209438 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.209702 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.209791 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.211157 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:10.211161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:10.211164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:10.211179 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:10.211206 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:10.211216 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:10.211289 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.211534 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.211615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:50:10.212795 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.204067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:36251 (local address 127.28.248.131:36315)
0928 02:50:10.204823 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.204869 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.206221 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.206765 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.206772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.206856 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:10.207353 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.207367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.208206 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.208209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.209277 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.209302 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.210001 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.210004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.210153 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.210768 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:50:10.210770 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:10.210836 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:50:10.210927 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:10.211362 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:10.211372 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:10.211378 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.211525 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.211833 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260928 02:50:10.213791 17394 heartbeater.cc:347] Connected to a master server at 127.28.248.190:36251
I20260928 02:50:10.214026 17394 heartbeater.cc:464] Registering TS with master...
I20260928 02:50:10.214532 17394 heartbeater.cc:511] Master 127.28.248.190:36251 requested a full tablet report, sending...
I20260928 02:50:10.215504 16944 ts_manager.cc:194] Registered new tserver with Master: 2026c7946c1641c7b4159248fb69f65d (127.28.248.131:41109)
I20260928 02:50:10.216022 16944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.248.131:36315
I20260928 02:50:10.225001 29666 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:50:10.230268 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.226888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.28.248.190:36251)
0928 02:50:10.227019 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.227023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.227183 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:50:10.227276 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.227278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.227317 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.227381 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:10.227597 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.228092 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.228932 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.229131 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.229203 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.229889 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:10.229892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:10.229894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:10.229904 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:10.229921 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:10.229928 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:10.229983 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.230056 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.230100 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:10.232880 16944 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:50:10.235987 16944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:50:10.250340 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.244240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:35623 (local address 127.0.0.1:38320)
0928 02:50:10.245012 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.245062 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.245189 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.246044 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.246047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.246064 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:10.246187 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.246194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.247611 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.247614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.248966 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.248978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.249778 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.249781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.249881 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.249917 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.250024 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.250192 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":74,"thread_start_us":78,"threads_started":1}
I20260928 02:50:10.250344 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.244861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.28.248.129:35623)
0928 02:50:10.245517 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.245531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.245555 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:50:10.245633 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.245641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.245862 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.246047 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:10.246296 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.247423 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.249126 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.249649 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.249779 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.249960 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.250052 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.250157 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":85,"threads_started":1}
I20260928 02:50:10.250385 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.244146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38977 (local address 127.0.0.1:50390)
0928 02:50:10.245014 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.245062 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.245199 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.245843 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.245850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.245871 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:10.246033 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.246041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.247514 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.247517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.248794 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.248803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.249873 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.249880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.249979 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.250002 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.250051 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.250197 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":88,"spinlock_wait_cycles":36224,"thread_start_us":70,"threads_started":1}
I20260928 02:50:10.250916 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.244837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.28.248.131:41109)
0928 02:50:10.245454 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.245464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.245774 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:50:10.245851 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.245856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.246046 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.246195 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:10.246534 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.247267 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.249088 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.249832 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.249954 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.250131 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.250640 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.250746 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":140,"threads_started":1}
I20260928 02:50:10.250550 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.244523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41109 (local address 127.0.0.1:34470)
0928 02:50:10.245585 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.245598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.245678 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.246336 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.246338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.246351 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:10.246427 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.246431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.247386 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.247390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.248973 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.248979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.250193 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.250196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.250280 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.250299 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.250366 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.250445 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":165,"thread_start_us":105,"threads_started":1}
I20260928 02:50:10.251225 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.244278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.28.248.130:38977)
0928 02:50:10.245322 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.245336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.245361 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:50:10.245437 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.245443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.245680 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.245839 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:10.246166 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.247410 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.248919 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.249723 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.250502 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.250676 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.250720 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.250795 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":126,"threads_started":1}
I20260928 02:50:10.254021 17197 tablet_service.cc:1467] Processing CreateTablet for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e (DEFAULT_TABLE table=test-table [id=2b039cf6bd1d437d9a06eb796b86e242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:10.254053 17328 tablet_service.cc:1467] Processing CreateTablet for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e (DEFAULT_TABLE table=test-table [id=2b039cf6bd1d437d9a06eb796b86e242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:10.254055 17066 tablet_service.cc:1467] Processing CreateTablet for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e (DEFAULT_TABLE table=test-table [id=2b039cf6bd1d437d9a06eb796b86e242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:50:10.254974 17197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:10.255033 17066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:10.255033 17328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:10.261345 17411 tablet_bootstrap.cc:492] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Bootstrap starting.
I20260928 02:50:10.262769 17413 tablet_bootstrap.cc:492] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: Bootstrap starting.
I20260928 02:50:10.263586 17411 tablet_bootstrap.cc:654] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:10.264313 17411 log.cc:824] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:10.264532 17413 tablet_bootstrap.cc:654] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:10.265075 17412 tablet_bootstrap.cc:492] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Bootstrap starting.
I20260928 02:50:10.265419 17413 log.cc:824] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:10.266551 17411 tablet_bootstrap.cc:492] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: No bootstrap required, opened a new log
I20260928 02:50:10.266763 17411 ts_tablet_manager.cc:1397] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:50:10.267107 17412 tablet_bootstrap.cc:654] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:10.267408 17413 tablet_bootstrap.cc:492] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: No bootstrap required, opened a new log
I20260928 02:50:10.267590 17413 ts_tablet_manager.cc:1397] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:50:10.267953 17412 log.cc:824] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:10.269937 17412 tablet_bootstrap.cc:492] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: No bootstrap required, opened a new log
I20260928 02:50:10.270118 17412 ts_tablet_manager.cc:1397] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:50:10.270521 17411 raft_consensus.cc:348] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.270762 17411 raft_consensus.cc:374] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:10.270820 17411 raft_consensus.cc:729] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31398fc0bd248f39a6f3322740c974a, State: Initialized, Role: FOLLOWER
I20260928 02:50:10.271294 17411 consensus_queue.cc:260] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.271924 17263 heartbeater.cc:503] Master 127.28.248.190:36251 was elected leader, sending a full tablet report...
I20260928 02:50:10.272284 17411 ts_tablet_manager.cc:1428] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 02:50:10.272644 17413 raft_consensus.cc:348] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.272904 17413 raft_consensus.cc:374] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:10.272969 17413 raft_consensus.cc:729] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52354de5acf4b69a249806fe294fa89, State: Initialized, Role: FOLLOWER
I20260928 02:50:10.273406 17413 consensus_queue.cc:260] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.274348 17413 ts_tablet_manager.cc:1428] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260928 02:50:10.274266 17412 raft_consensus.cc:348] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.274530 17412 raft_consensus.cc:374] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:10.274590 17412 raft_consensus.cc:729] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2026c7946c1641c7b4159248fb69f65d, State: Initialized, Role: FOLLOWER
I20260928 02:50:10.274871 17132 heartbeater.cc:503] Master 127.28.248.190:36251 was elected leader, sending a full tablet report...
I20260928 02:50:10.275071 17412 consensus_queue.cc:260] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.275707 17394 heartbeater.cc:503] Master 127.28.248.190:36251 was elected leader, sending a full tablet report...
I20260928 02:50:10.276870 17412 ts_tablet_manager.cc:1428] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 02:50:10.309441 17418 raft_consensus.cc:482] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:50:10.309651 17418 raft_consensus.cc:504] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.310828 17418 leader_election.cc:288] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c31398fc0bd248f39a6f3322740c974a (127.28.248.130:38977), 2026c7946c1641c7b4159248fb69f65d (127.28.248.131:41109)
I20260928 02:50:10.315623 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.311128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:44341 (local address 127.28.248.130:38977)
0928 02:50:10.311262 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.311266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.311280 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:10.311447 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.311450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.311497 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.311598 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:10.311905 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.312708 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.314321 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.314925 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.315036 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.315083 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.315406 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.315483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:10.315657 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.310998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.130:38977 (local address 127.28.248.129:44341)
0928 02:50:10.311187 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.311202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.311291 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.311622 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.311628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.311645 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:10.311752 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.311775 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.312839 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.312844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.314198 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.314208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.315051 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.315054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.315237 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.315301 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.315377 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.315488 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":153}
I20260928 02:50:10.315666 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.311107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.131:41109 (local address 127.28.248.129:35297)
0928 02:50:10.311470 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.311480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:10.311554 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:10.311812 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:10.311818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:10.311837 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:50:10.311952 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.311956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.312866 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.312868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.314198 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:10.314208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.315049 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.315052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:10.315233 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.315298 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:50:10.315377 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:50:10.315488 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":113,"thread_start_us":122,"threads_started":1}
I20260928 02:50:10.316015 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.311192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.248.129:35297 (local address 127.28.248.131:41109)
0928 02:50:10.311326 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.311331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.311562 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:50:10.311664 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.311667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.311720 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.311777 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:50:10.312053 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.312780 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.314321 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.314919 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.315036 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.315083 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.315821 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.315910 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:50:10.316310 17217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d07c23f1e24053bf0a6c9c2cc45a3e" candidate_uuid: "a52354de5acf4b69a249806fe294fa89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31398fc0bd248f39a6f3322740c974a" is_pre_election: true
I20260928 02:50:10.316596 17217 raft_consensus.cc:2454] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a52354de5acf4b69a249806fe294fa89 in term 0.
I20260928 02:50:10.316574 17348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d07c23f1e24053bf0a6c9c2cc45a3e" candidate_uuid: "a52354de5acf4b69a249806fe294fa89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2026c7946c1641c7b4159248fb69f65d" is_pre_election: true
I20260928 02:50:10.316890 17348 raft_consensus.cc:2454] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a52354de5acf4b69a249806fe294fa89 in term 0.
I20260928 02:50:10.317102 17022 leader_election.cc:302] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a52354de5acf4b69a249806fe294fa89, c31398fc0bd248f39a6f3322740c974a; no voters:
I20260928 02:50:10.317354 17418 raft_consensus.cc:2781] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:50:10.317453 17418 raft_consensus.cc:482] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:50:10.317512 17418 raft_consensus.cc:3037] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:10.318498 17418 raft_consensus.cc:504] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.318881 17418 leader_election.cc:288] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [CANDIDATE]: Term 1 election: Requested vote from peers c31398fc0bd248f39a6f3322740c974a (127.28.248.130:38977), 2026c7946c1641c7b4159248fb69f65d (127.28.248.131:41109)
I20260928 02:50:10.319147 17217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d07c23f1e24053bf0a6c9c2cc45a3e" candidate_uuid: "a52354de5acf4b69a249806fe294fa89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c31398fc0bd248f39a6f3322740c974a"
I20260928 02:50:10.319227 17348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d07c23f1e24053bf0a6c9c2cc45a3e" candidate_uuid: "a52354de5acf4b69a249806fe294fa89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2026c7946c1641c7b4159248fb69f65d"
I20260928 02:50:10.319275 17217 raft_consensus.cc:3037] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:10.319332 17348 raft_consensus.cc:3037] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:10.320531 17348 raft_consensus.cc:2454] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a52354de5acf4b69a249806fe294fa89 in term 1.
I20260928 02:50:10.320531 17217 raft_consensus.cc:2454] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a52354de5acf4b69a249806fe294fa89 in term 1.
I20260928 02:50:10.320880 17019 leader_election.cc:302] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2026c7946c1641c7b4159248fb69f65d, a52354de5acf4b69a249806fe294fa89; no voters:
I20260928 02:50:10.321066 17418 raft_consensus.cc:2781] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:10.321311 17418 raft_consensus.cc:686] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 1 LEADER]: Becoming Leader. State: Replica: a52354de5acf4b69a249806fe294fa89, State: Running, Role: LEADER
I20260928 02:50:10.321652 17418 consensus_queue.cc:237] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } }
I20260928 02:50:10.324678 16943 catalog_manager.cc:5495] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 reported cstate change: term changed from 0 to 1, leader changed from <none> to a52354de5acf4b69a249806fe294fa89 (127.28.248.129). New cstate: current_term: 1 leader_uuid: "a52354de5acf4b69a249806fe294fa89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a52354de5acf4b69a249806fe294fa89" member_type: VOTER last_known_addr { host: "127.28.248.129" port: 35623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 } health_report { overall_health: UNKNOWN } } }
W20260928 02:50:10.338217 17133 tablet.cc:2369] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:50:10.346251 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.343013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.28.248.129:35623)
0928 02:50:10.343120 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:10.343123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:10.343135 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:50:10.343252 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:10.343255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:10.343319 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:10.343401 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:10.343604 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.344118 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.344991 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:10.345197 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:10.345273 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:10.345522 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:10.345910 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:10.345997 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:10.346047 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:50:10.346091 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:50:10.351768 17348 raft_consensus.cc:1261] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 1 FOLLOWER]: Refusing update from remote peer a52354de5acf4b69a249806fe294fa89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:10.351768 17217 raft_consensus.cc:1261] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 1 FOLLOWER]: Refusing update from remote peer a52354de5acf4b69a249806fe294fa89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:10.352456 17421 consensus_queue.cc:1035] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2026c7946c1641c7b4159248fb69f65d" member_type: VOTER last_known_addr { host: "127.28.248.131" port: 41109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:10.352645 17418 consensus_queue.cc:1035] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31398fc0bd248f39a6f3322740c974a" member_type: VOTER last_known_addr { host: "127.28.248.130" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:10.368741 16943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:10.368932 16943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:50:10.371353 16943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 700dc00ac7074b449edad57aa18509fc: Sending DeleteTablet for 3 replicas of tablet b3d07c23f1e24053bf0a6c9c2cc45a3e
I20260928 02:50:10.372109 17066 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:10 UTC) from {username='slave'} at 127.0.0.1:38320
I20260928 02:50:10.372105 17197 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:10 UTC) from {username='slave'} at 127.0.0.1:50390
I20260928 02:50:10.372263 17328 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:50:10 UTC) from {username='slave'} at 127.0.0.1:34470
I20260928 02:50:10.372800 17434 tablet_replica.cc:330] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89: stopping tablet replica
I20260928 02:50:10.373046 17436 tablet_replica.cc:330] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: stopping tablet replica
I20260928 02:50:10.373051 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17004
I20260928 02:50:10.373137 17434 raft_consensus.cc:2229] T b3d07c23f1e24053bf0a6c9c2cc45a3e P a52354de5acf4b69a249806fe294fa89 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:50:10.373181 17435 tablet_replica.cc:330] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: stopping tablet replica
I20260928 02:50:10.373809 17436 raft_consensus.cc:2229] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:10.374140 17436 raft_consensus.cc:2258] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:10.373543 17435 raft_consensus.cc:2229] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:50:10.374634 17435 raft_consensus.cc:2258] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:50:10.376276 17435 ts_tablet_manager.cc:1905] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:10.376760 17436 ts_tablet_manager.cc:1905] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:50:10.378916 17436 ts_tablet_manager.cc:1918] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:10.379026 17436 log.cc:1196] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-2/wal/wals/b3d07c23f1e24053bf0a6c9c2cc45a3e
I20260928 02:50:10.379285 17436 ts_tablet_manager.cc:1939] T b3d07c23f1e24053bf0a6c9c2cc45a3e P 2026c7946c1641c7b4159248fb69f65d: Deleting consensus metadata
I20260928 02:50:10.379810 17435 ts_tablet_manager.cc:1918] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:50:10.379912 17435 log.cc:1196] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Deleting WAL directory at /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563560542449-29666-0/minicluster-data/ts-1/wal/wals/b3d07c23f1e24053bf0a6c9c2cc45a3e
I20260928 02:50:10.380235 17435 ts_tablet_manager.cc:1939] T b3d07c23f1e24053bf0a6c9c2cc45a3e P c31398fc0bd248f39a6f3322740c974a: Deleting consensus metadata
I20260928 02:50:10.380481 16928 catalog_manager.cc:4841] TS 2026c7946c1641c7b4159248fb69f65d (127.28.248.131:41109): tablet b3d07c23f1e24053bf0a6c9c2cc45a3e (table test-table [id=2b039cf6bd1d437d9a06eb796b86e242]) successfully deleted
W20260928 02:50:10.381153 16929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.248.129:35623 (error 108)
I20260928 02:50:10.381404 16931 catalog_manager.cc:4841] TS c31398fc0bd248f39a6f3322740c974a (127.28.248.130:38977): tablet b3d07c23f1e24053bf0a6c9c2cc45a3e (table test-table [id=2b039cf6bd1d437d9a06eb796b86e242]) successfully deleted
I20260928 02:50:10.382015 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17135
I20260928 02:50:10.382264 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:10.381935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.129:35623 (local address 127.0.0.1:38350)
0928 02:50:10.382065 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:10.382167 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.129:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:50:10.382438 16929 catalog_manager.cc:4584] TS a52354de5acf4b69a249806fe294fa89 (127.28.248.129:35623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3d07c23f1e24053bf0a6c9c2cc45a3e: Network error: Client connection negotiation failed: client connection to 127.28.248.129:35623: connect: Connection refused (error 111)
I20260928 02:50:10.388481 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17266
I20260928 02:50:10.394088 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 16913
2026-09-28T02:50:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1033 ms)
[----------] 6 tests from EncryptionPolicyTest (10727 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:10 dist-test-slave-r238 krb5kdc[17440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:10 dist-test-slave-r238 krb5kdc[17440](info): set up 2 sockets
Sep 28 02:50:10 dist-test-slave-r238 krb5kdc[17440](info): commencing operation
krb5kdc: starting...
W20260928 02:50:11.472901 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:11 dist-test-slave-r238 krb5kdc[17440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:11.593807 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:11.700139 17453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:11.700385 17453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:11.700424 17453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:11.704236 17453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:11.704288 17453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:11.704306 17453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:11.704322 17453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:11.704337 17453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:11.709344 17453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:11.710289 17453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:11.711259 17453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:11.719791 17461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:11.719806 17458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:11.719817 17459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:11.720196 17453 server_base.cc:1034] running on GCE node
I20260928 02:50:11.720759 17453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:11.721841 17453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:11.723029 17453 hybrid_clock.cc:648] HybridClock initialized: now 1790563811723003 us; error 39 us; skew 500 ppm
Sep 28 02:50:11 dist-test-slave-r238 krb5kdc[17440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:11.726094 17453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:11.727182 17453 webserver.cc:460] Webserver started at http://127.0.0.1:34275/ using document root <none> and password file <none>
I20260928 02:50:11.727861 17453 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:11.727947 17453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:11.728185 17453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:11.730365 17453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "3ddc99a35c22468ea3836b9b1bf7b86c"
format_stamp: "Formatted at 2026-09-28 02:50:11 on dist-test-slave-r238"
server_key: "1595e3fe8425c92298b9395160e2456b"
server_key_iv: "66f761fdf2c958b0b5c47ec6be572dca"
server_key_version: "encryptionkey@0"
I20260928 02:50:11.730880 17453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "3ddc99a35c22468ea3836b9b1bf7b86c"
format_stamp: "Formatted at 2026-09-28 02:50:11 on dist-test-slave-r238"
server_key: "1595e3fe8425c92298b9395160e2456b"
server_key_iv: "66f761fdf2c958b0b5c47ec6be572dca"
server_key_version: "encryptionkey@0"
I20260928 02:50:11.734448 17453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:50:11.736720 17468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:11.737902 17453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:11.738029 17453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "3ddc99a35c22468ea3836b9b1bf7b86c"
format_stamp: "Formatted at 2026-09-28 02:50:11 on dist-test-slave-r238"
server_key: "1595e3fe8425c92298b9395160e2456b"
server_key_iv: "66f761fdf2c958b0b5c47ec6be572dca"
server_key_version: "encryptionkey@0"
I20260928 02:50:11.738138 17453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:11.762607 17453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:11.765565 17453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:11.765761 17453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:11.772408 17453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40659
I20260928 02:50:11.772473 17520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40659 every 8 connection(s)
I20260928 02:50:11.773388 17453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:11.776373 17521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:11.779564 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17453
I20260928 02:50:11.779717 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:11.780050 29666 external_mini_cluster.cc:1428] Setting key 3fbfc9d4ae0fe308b293137b4ac86f41
I20260928 02:50:11.783082 17521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c: Bootstrap starting.
I20260928 02:50:11.785897 17521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:11.786692 17521 log.cc:824] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:11.788779 17521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c: No bootstrap required, opened a new log
Sep 28 02:50:11 dist-test-slave-r238 krb5kdc[17440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:50:11.793128 17521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260928 02:50:11.793426 17521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:11.793495 17521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddc99a35c22468ea3836b9b1bf7b86c, State: Initialized, Role: FOLLOWER
I20260928 02:50:11.794035 17521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260928 02:50:11.794222 17521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:11.794302 17521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:11.794411 17521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:11.795629 17521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260928 02:50:11.796032 17521 leader_election.cc:302] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ddc99a35c22468ea3836b9b1bf7b86c; no voters:
I20260928 02:50:11.796342 17521 leader_election.cc:288] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:11.796445 17526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:11.796512 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:11.781460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42407 (local address 127.0.0.1:40659)
0928 02:50:11.781887 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:11.781908 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:11.781954 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:50:11.782810 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:11.782834 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:11.783294 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:11.783666 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:11.784324 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:11.785991 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:11.786873 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:11.787543 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:11.787706 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:11.790180 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:11.790187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:11.790204 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:11.790248 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:11.793044 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:11.793593 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:11.793599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:11.793605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:11.793750 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:11.794098 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:11.794101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:11.794103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:11.794763 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:11.795022 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:11.795249 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:50:11.795493 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":126,"threads_started":1}
I20260928 02:50:11.796712 17526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [term 1 LEADER]: Becoming Leader. State: Replica: 3ddc99a35c22468ea3836b9b1bf7b86c, State: Running, Role: LEADER
I20260928 02:50:11.797049 17526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } }
I20260928 02:50:11.797560 17521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:11.798533 17528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ddc99a35c22468ea3836b9b1bf7b86c. Latest consensus state: current_term: 1 leader_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } } }
I20260928 02:50:11.798646 17528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:11.799510 17535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:11.799804 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:11.799960 17527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddc99a35c22468ea3836b9b1bf7b86c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40659 } } }
I20260928 02:50:11.800760 17527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:11.802815 17535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:11.807981 17535 catalog_manager.cc:1347] Generated new cluster ID: f737fba731ba4f7992cff73d7a08a693
I20260928 02:50:11.808038 17535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:11.813030 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:11.801972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51997 (local address 127.0.0.1:40659)
0928 02:50:11.802075 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:11.802079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:11.804658 (+ 2579us) server_negotiation.cc:407] Connection header received
0928 02:50:11.804816 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:11.804820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:11.804866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:11.804952 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:11.805718 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:11.806938 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:11.808014 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:11.808294 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:11.808406 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:11.809865 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:11.809869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:11.809872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:11.809900 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:11.811306 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:11.811780 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:11.811784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:11.811786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:11.811860 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:11.812173 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:11.812176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:11.812178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:11.812343 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:11.812507 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:11.812782 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:50:11.812872 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:50:11.814532 17535 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:11.815645 17535 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:11.822844 17535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ddc99a35c22468ea3836b9b1bf7b86c: Generated new TSK 0
I20260928 02:50:11.823529 17535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:11.830399 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17453
W20260928 02:50:11.835755 17438 connection.cc:535] client connection to 127.0.0.1:40659 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40659): Connection reset by peer (error 104)
2026-09-28T02:50:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:11 dist-test-slave-r238 krb5kdc[17552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:11 dist-test-slave-r238 krb5kdc[17552](info): set up 2 sockets
Sep 28 02:50:11 dist-test-slave-r238 krb5kdc[17552](info): commencing operation
krb5kdc: starting...
W20260928 02:50:12.864988 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:12 dist-test-slave-r238 krb5kdc[17552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:12.999706 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:13.111941 17565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:13.112272 17565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:13.112349 17565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:13.116456 17565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:13.116541 17565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:13.116582 17565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:13.116644 17565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:13.116695 17565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:13.122032 17565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:13.123095 17565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:13.124097 17565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:13.132740 17573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:13.132740 17571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:13.132802 17570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:13.132951 17565 server_base.cc:1034] running on GCE node
I20260928 02:50:13.133736 17565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:13.134863 17565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:13.136034 17565 hybrid_clock.cc:648] HybridClock initialized: now 1790563813136005 us; error 40 us; skew 500 ppm
Sep 28 02:50:13 dist-test-slave-r238 krb5kdc[17552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:13.138753 17565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:13.139659 17565 webserver.cc:460] Webserver started at http://127.0.0.1:41925/ using document root <none> and password file <none>
I20260928 02:50:13.140166 17565 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:13.140223 17565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:13.140403 17565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:13.142419 17565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "6f075cd95fd2492283d97d973fef8690"
format_stamp: "Formatted at 2026-09-28 02:50:13 on dist-test-slave-r238"
server_key: "0c69f92fa40fafa041bcd7580dc654e7"
server_key_iv: "bf51146a911886adaa5eec1d7f3c40c6"
server_key_version: "encryptionkey@0"
I20260928 02:50:13.142937 17565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "6f075cd95fd2492283d97d973fef8690"
format_stamp: "Formatted at 2026-09-28 02:50:13 on dist-test-slave-r238"
server_key: "0c69f92fa40fafa041bcd7580dc654e7"
server_key_iv: "bf51146a911886adaa5eec1d7f3c40c6"
server_key_version: "encryptionkey@0"
I20260928 02:50:13.147102 17565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:50:13.149617 17580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:13.151011 17565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:13.151117 17565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "6f075cd95fd2492283d97d973fef8690"
format_stamp: "Formatted at 2026-09-28 02:50:13 on dist-test-slave-r238"
server_key: "0c69f92fa40fafa041bcd7580dc654e7"
server_key_iv: "bf51146a911886adaa5eec1d7f3c40c6"
server_key_version: "encryptionkey@0"
I20260928 02:50:13.151217 17565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:13.191547 17565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:13.194528 17565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:13.194682 17565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:13.202142 17565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45677
I20260928 02:50:13.202167 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45677 every 8 connection(s)
I20260928 02:50:13.203097 17565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:13.205942 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:13.206516 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17565
I20260928 02:50:13.206647 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:13.206950 29666 external_mini_cluster.cc:1428] Setting key 2643d3058e25858a6b96fd7227ec7ecd
I20260928 02:50:13.212754 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690: Bootstrap starting.
I20260928 02:50:13.215569 17633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:13.216496 17633 log.cc:824] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:13.217453 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:13.208366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54757 (local address 127.0.0.1:45677)
0928 02:50:13.208861 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:13.208880 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:13.208925 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:50:13.209787 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:13.209811 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:13.210235 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:13.210611 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:13.211045 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:13.212524 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:13.213489 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:13.214183 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:13.214339 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:13.215384 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:13.215393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:13.215412 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:13.215965 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:13.216031 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:13.216060 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:13.216191 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:13.216365 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:50:13.216505 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":115,"threads_started":1}
I20260928 02:50:13.218680 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690: No bootstrap required, opened a new log
I20260928 02:50:13.220350 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:13.222755 17633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f075cd95fd2492283d97d973fef8690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45677 } }
I20260928 02:50:13.223047 17633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:13.223131 17633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f075cd95fd2492283d97d973fef8690, State: Initialized, Role: FOLLOWER
I20260928 02:50:13.223620 17633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f075cd95fd2492283d97d973fef8690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45677 } }
I20260928 02:50:13.223773 17633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:13.223832 17633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:13.223920 17633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:13.224953 17633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f075cd95fd2492283d97d973fef8690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45677 } }
I20260928 02:50:13.225333 17633 leader_election.cc:302] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f075cd95fd2492283d97d973fef8690; no voters:
I20260928 02:50:13.225718 17633 leader_election.cc:288] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:13.226089 17643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:13.226269 17643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [term 1 LEADER]: Becoming Leader. State: Replica: 6f075cd95fd2492283d97d973fef8690, State: Running, Role: LEADER
I20260928 02:50:13.226558 17643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f075cd95fd2492283d97d973fef8690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45677 } }
I20260928 02:50:13.226867 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:13.222386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40687 (local address 127.0.0.1:45677)
0928 02:50:13.222498 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:13.222502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:13.222719 (+ 217us) server_negotiation.cc:407] Connection header received
0928 02:50:13.222797 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:13.222801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:13.222856 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:13.222933 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:13.223217 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:13.223978 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:13.224886 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:13.225227 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:13.225334 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:13.226314 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:13.226318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:13.226322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:13.226337 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:13.226366 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:13.226378 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:13.226441 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:13.226690 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:50:13.226723 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:50:13.227072 17633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:13.228199 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f075cd95fd2492283d97d973fef8690. Latest consensus state: current_term: 1 leader_uuid: "6f075cd95fd2492283d97d973fef8690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f075cd95fd2492283d97d973fef8690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45677 } } }
I20260928 02:50:13.228312 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:13.228690 17652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:13.228789 17644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f075cd95fd2492283d97d973fef8690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f075cd95fd2492283d97d973fef8690" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45677 } } }
I20260928 02:50:13.228891 17644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:13.231719 17652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:13.237030 17652 catalog_manager.cc:1347] Generated new cluster ID: f8e4871170df4c63aa942fd6fb7d1d66
I20260928 02:50:13.237113 17652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:13.258183 17652 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:13.259186 17652 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:13.267097 17652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f075cd95fd2492283d97d973fef8690: Generated new TSK 0
I20260928 02:50:13.267894 17652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:13.283075 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17565
W20260928 02:50:13.288638 17550 connection.cc:535] client connection to 127.0.0.1:45677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45677): Connection reset by peer (error 104)
2026-09-28T02:50:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:13 dist-test-slave-r238 krb5kdc[17664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:13 dist-test-slave-r238 krb5kdc[17664](info): set up 2 sockets
Sep 28 02:50:13 dist-test-slave-r238 krb5kdc[17664](info): commencing operation
krb5kdc: starting...
W20260928 02:50:14.321357 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:14 dist-test-slave-r238 krb5kdc[17664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:14.448037 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:14.561776 17677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:14.562156 17677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:14.562247 17677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:14.566784 17677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:14.566895 17677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:14.566931 17677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:14.566969 17677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:14.567009 17677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:14.572819 17677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:14.574036 17677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:14.575388 17677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:14.584542 17683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:14.584519 17682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:14.584522 17685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:14.584894 17677 server_base.cc:1034] running on GCE node
I20260928 02:50:14.585453 17677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:14.586614 17677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:14.587773 17677 hybrid_clock.cc:648] HybridClock initialized: now 1790563814587758 us; error 31 us; skew 500 ppm
Sep 28 02:50:14 dist-test-slave-r238 krb5kdc[17664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:14.590598 17677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:14.591504 17677 webserver.cc:460] Webserver started at http://127.0.0.1:33639/ using document root <none> and password file <none>
I20260928 02:50:14.592069 17677 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:14.592130 17677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:14.592300 17677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:14.594462 17677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "ecb9a5a7390b4885aceafc81e6e9761d"
format_stamp: "Formatted at 2026-09-28 02:50:14 on dist-test-slave-r238"
server_key: "bb56922fb80aff4ad89d7a00585e878c"
server_key_iv: "4c8c9e03e581d567d0f6d1d7673ec669"
server_key_version: "encryptionkey@0"
I20260928 02:50:14.595003 17677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "ecb9a5a7390b4885aceafc81e6e9761d"
format_stamp: "Formatted at 2026-09-28 02:50:14 on dist-test-slave-r238"
server_key: "bb56922fb80aff4ad89d7a00585e878c"
server_key_iv: "4c8c9e03e581d567d0f6d1d7673ec669"
server_key_version: "encryptionkey@0"
I20260928 02:50:14.598846 17677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:50:14.601802 17692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:14.603293 17677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:14.603446 17677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "ecb9a5a7390b4885aceafc81e6e9761d"
format_stamp: "Formatted at 2026-09-28 02:50:14 on dist-test-slave-r238"
server_key: "bb56922fb80aff4ad89d7a00585e878c"
server_key_iv: "4c8c9e03e581d567d0f6d1d7673ec669"
server_key_version: "encryptionkey@0"
I20260928 02:50:14.603585 17677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:14.608991 17677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:14.612354 17677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:14.612601 17677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:14.620766 17677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37065
I20260928 02:50:14.620795 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37065 every 8 connection(s)
I20260928 02:50:14.621887 17677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:14.624878 17745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:14.625070 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17677
I20260928 02:50:14.625247 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:14.625590 29666 external_mini_cluster.cc:1428] Setting key 917cb8059220d560f2b7502a7274ada6
I20260928 02:50:14.631623 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:14.627128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48271 (local address 127.0.0.1:37065)
0928 02:50:14.627603 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:14.627623 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:14.627670 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:50:14.628466 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:14.628492 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:14.628934 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:14.629321 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:14.630135 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:14.630140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:14.630155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:14.630167 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:14.630231 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:14.630258 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:14.630403 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:14.630471 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:50:14.630635 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":146,"threads_started":1}
I20260928 02:50:14.632763 17745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d: Bootstrap starting.
I20260928 02:50:14.634615 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:14.635275 17745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:14.636085 17745 log.cc:824] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:14.638214 17745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d: No bootstrap required, opened a new log
I20260928 02:50:14.641745 17745 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260928 02:50:14.642011 17745 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:14.642072 17745 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb9a5a7390b4885aceafc81e6e9761d, State: Initialized, Role: FOLLOWER
I20260928 02:50:14.642531 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:14.636727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60037 (local address 127.0.0.1:37065)
0928 02:50:14.636905 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:14.636910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:14.640981 (+ 4071us) server_negotiation.cc:407] Connection header received
0928 02:50:14.641152 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:14.641156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:14.641194 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:14.641277 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:14.642037 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:14.642041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:14.642044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:14.642056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:14.642087 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:14.642102 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:14.642183 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:14.642339 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:50:14.642387 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:50:14.642669 17745 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260928 02:50:14.642851 17745 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:14.642925 17745 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:14.643033 17745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:14.644124 17745 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260928 02:50:14.644484 17745 leader_election.cc:302] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecb9a5a7390b4885aceafc81e6e9761d; no voters:
I20260928 02:50:14.644834 17745 leader_election.cc:288] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:14.644948 17755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:14.645162 17755 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [term 1 LEADER]: Becoming Leader. State: Replica: ecb9a5a7390b4885aceafc81e6e9761d, State: Running, Role: LEADER
I20260928 02:50:14.645517 17755 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } }
I20260928 02:50:14.646188 17745 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:14.646755 17755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecb9a5a7390b4885aceafc81e6e9761d. Latest consensus state: current_term: 1 leader_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } } }
I20260928 02:50:14.646926 17755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:14.647219 17756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9a5a7390b4885aceafc81e6e9761d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37065 } } }
I20260928 02:50:14.647334 17762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:14.647359 17756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:14.650799 17762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:14.656375 17762 catalog_manager.cc:1347] Generated new cluster ID: 84b28b46470d4bf0b7b71767654b13ed
I20260928 02:50:14.656462 17762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:14.683995 17762 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:14.684387 17762 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:14.689683 17762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecb9a5a7390b4885aceafc81e6e9761d: Generated new TSK 0
I20260928 02:50:14.690418 17762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:14.700189 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17677
2026-09-28T02:50:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:14 dist-test-slave-r238 krb5kdc[17776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:14 dist-test-slave-r238 krb5kdc[17776](info): set up 2 sockets
Sep 28 02:50:14 dist-test-slave-r238 krb5kdc[17776](info): commencing operation
krb5kdc: starting...
W20260928 02:50:15.768939 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:15 dist-test-slave-r238 krb5kdc[17776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:15.892045 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:16.008219 17789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:16.008489 17789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:16.008533 17789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:16.012539 17789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:16.012629 17789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:16.012651 17789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:16.012667 17789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:16.012683 17789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:16.018066 17789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:16.019160 17789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:16.020282 17789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:16.028901 17795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:16.028983 17794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:16.028872 17797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:16.029148 17789 server_base.cc:1034] running on GCE node
I20260928 02:50:16.029668 17789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:16.030745 17789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:16.031929 17789 hybrid_clock.cc:648] HybridClock initialized: now 1790563816031901 us; error 40 us; skew 500 ppm
Sep 28 02:50:16 dist-test-slave-r238 krb5kdc[17776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:16.034701 17789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:16.035629 17789 webserver.cc:460] Webserver started at http://127.0.0.1:43809/ using document root <none> and password file <none>
I20260928 02:50:16.036177 17789 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:16.036281 17789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:16.036501 17789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:16.038549 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "6949eab28a6348439681fdbb0a1269b7"
format_stamp: "Formatted at 2026-09-28 02:50:16 on dist-test-slave-r238"
server_key: "e277e88d6febae8c8f7e6efb94c30fbc"
server_key_iv: "667730a2d139490ee992f0e382638925"
server_key_version: "encryptionkey@0"
I20260928 02:50:16.039111 17789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "6949eab28a6348439681fdbb0a1269b7"
format_stamp: "Formatted at 2026-09-28 02:50:16 on dist-test-slave-r238"
server_key: "e277e88d6febae8c8f7e6efb94c30fbc"
server_key_iv: "667730a2d139490ee992f0e382638925"
server_key_version: "encryptionkey@0"
I20260928 02:50:16.042868 17789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:50:16.045370 17804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:16.046628 17789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:16.046752 17789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "6949eab28a6348439681fdbb0a1269b7"
format_stamp: "Formatted at 2026-09-28 02:50:16 on dist-test-slave-r238"
server_key: "e277e88d6febae8c8f7e6efb94c30fbc"
server_key_iv: "667730a2d139490ee992f0e382638925"
server_key_version: "encryptionkey@0"
I20260928 02:50:16.046850 17789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:16.061136 17789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:16.064107 17789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:16.064329 17789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:16.071930 17789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44317
I20260928 02:50:16.071969 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44317 every 8 connection(s)
I20260928 02:50:16.072988 17789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:16.075752 17857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:16.078959 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17789
I20260928 02:50:16.079151 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:16.079487 29666 external_mini_cluster.cc:1428] Setting key c85dc2a745c184a6a55444d1bee92596
I20260928 02:50:16.082492 17857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7: Bootstrap starting.
I20260928 02:50:16.085204 17857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:16.086135 17857 log.cc:824] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:16.088498 17857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7: No bootstrap required, opened a new log
Sep 28 02:50:16 dist-test-slave-r238 krb5kdc[17776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:50:16.092270 17857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6949eab28a6348439681fdbb0a1269b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44317 } }
I20260928 02:50:16.092517 17857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:16.092552 17857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6949eab28a6348439681fdbb0a1269b7, State: Initialized, Role: FOLLOWER
I20260928 02:50:16.093091 17857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6949eab28a6348439681fdbb0a1269b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44317 } }
I20260928 02:50:16.093271 17857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:16.093348 17857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:16.093459 17857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:16.094563 17857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6949eab28a6348439681fdbb0a1269b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44317 } }
I20260928 02:50:16.094964 17857 leader_election.cc:302] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6949eab28a6348439681fdbb0a1269b7; no voters:
I20260928 02:50:16.095328 17857 leader_election.cc:288] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:16.095448 17862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:16.095700 17862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6949eab28a6348439681fdbb0a1269b7, State: Running, Role: LEADER
I20260928 02:50:16.096091 17862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6949eab28a6348439681fdbb0a1269b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44317 } }
I20260928 02:50:16.096360 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:16.080886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36933 (local address 127.0.0.1:44317)
0928 02:50:16.081350 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:16.081385 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:16.081431 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:50:16.082260 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:16.082284 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:16.082734 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:16.083093 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:16.083762 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:16.085263 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:16.086486 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:16.087194 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:16.087331 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:16.090346 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:16.090351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:16.090365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:16.090391 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:16.092953 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:16.093472 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:16.093483 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:16.093487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:16.093647 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:16.093959 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:16.093961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:16.093962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:16.094380 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:16.094548 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:16.094897 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:50:16.095248 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260928 02:50:16.096683 17857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:16.097858 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6949eab28a6348439681fdbb0a1269b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6949eab28a6348439681fdbb0a1269b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44317 } } }
I20260928 02:50:16.098052 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:16.098364 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6949eab28a6348439681fdbb0a1269b7. Latest consensus state: current_term: 1 leader_uuid: "6949eab28a6348439681fdbb0a1269b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6949eab28a6348439681fdbb0a1269b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44317 } } }
I20260928 02:50:16.098524 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:16.099341 17872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:16.099766 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:16.101972 17872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:16.107918 17872 catalog_manager.cc:1347] Generated new cluster ID: 35621ea2932a4ed28eef8c67bf895860
I20260928 02:50:16.107993 17872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:16.111052 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:16.102065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51827 (local address 127.0.0.1:44317)
0928 02:50:16.102169 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:16.102174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:16.102518 (+ 344us) server_negotiation.cc:407] Connection header received
0928 02:50:16.102644 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:16.102647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:16.102695 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:16.102782 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:16.103653 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:16.104203 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:16.105092 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:16.105322 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:16.105437 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:16.106938 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:16.106942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:16.106945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:16.106974 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:16.109325 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:16.109827 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:16.109830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:16.109832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:16.109915 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:16.110214 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:16.110217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:16.110218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:16.110401 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:16.110512 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:16.110778 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:50:16.110891 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:50:16.134740 17872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:16.135690 17872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:16.142138 17872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6949eab28a6348439681fdbb0a1269b7: Generated new TSK 0
I20260928 02:50:16.142843 17872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:16.148203 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17789
2026-09-28T02:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:16 dist-test-slave-r238 krb5kdc[17888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:16 dist-test-slave-r238 krb5kdc[17888](info): set up 2 sockets
Sep 28 02:50:16 dist-test-slave-r238 krb5kdc[17888](info): commencing operation
krb5kdc: starting...
W20260928 02:50:17.244899 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:17 dist-test-slave-r238 krb5kdc[17888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:17.363494 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:17.471509 17901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:17.471753 17901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:17.471801 17901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:17.475744 17901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:17.475807 17901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:17.475827 17901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:17.475843 17901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:17.475858 17901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:17.480890 17901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:17.481803 17901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:17.482731 17901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:17.490990 17907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:17.491133 17906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:17.491219 17901 server_base.cc:1034] running on GCE node
W20260928 02:50:17.491379 17909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:17.491966 17901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:17.493233 17901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:17.494426 17901 hybrid_clock.cc:648] HybridClock initialized: now 1790563817494400 us; error 37 us; skew 500 ppm
Sep 28 02:50:17 dist-test-slave-r238 krb5kdc[17888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:17.497395 17901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:17.498348 17901 webserver.cc:460] Webserver started at http://127.0.0.1:40349/ using document root <none> and password file <none>
I20260928 02:50:17.498957 17901 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:17.499027 17901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:17.499248 17901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:17.501372 17901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "7bd1cd1a558b460990ceeae878286be1"
format_stamp: "Formatted at 2026-09-28 02:50:17 on dist-test-slave-r238"
server_key: "45b866079a86840a479a79793730d10d"
server_key_iv: "b06dda695e1f8966da13179c1272125b"
server_key_version: "encryptionkey@0"
I20260928 02:50:17.501958 17901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "7bd1cd1a558b460990ceeae878286be1"
format_stamp: "Formatted at 2026-09-28 02:50:17 on dist-test-slave-r238"
server_key: "45b866079a86840a479a79793730d10d"
server_key_iv: "b06dda695e1f8966da13179c1272125b"
server_key_version: "encryptionkey@0"
I20260928 02:50:17.505776 17901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:50:17.508102 17916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:17.509333 17901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:17.509473 17901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "7bd1cd1a558b460990ceeae878286be1"
format_stamp: "Formatted at 2026-09-28 02:50:17 on dist-test-slave-r238"
server_key: "45b866079a86840a479a79793730d10d"
server_key_iv: "b06dda695e1f8966da13179c1272125b"
server_key_version: "encryptionkey@0"
I20260928 02:50:17.509611 17901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:17.539208 17901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:17.542737 17901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:17.542958 17901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:17.550494 17901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45675
I20260928 02:50:17.550523 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45675 every 8 connection(s)
I20260928 02:50:17.551523 17901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:17.554261 17969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:17.559544 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 17901
I20260928 02:50:17.559695 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:17.559757 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1: Bootstrap starting.
I20260928 02:50:17.560025 29666 external_mini_cluster.cc:1428] Setting key 6f924c2db0acae206db053531d1afb27
I20260928 02:50:17.562515 17969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:17.563369 17969 log.cc:824] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:17.565635 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1: No bootstrap required, opened a new log
I20260928 02:50:17.569823 17969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd1cd1a558b460990ceeae878286be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260928 02:50:17.570097 17969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:17.570152 17969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd1cd1a558b460990ceeae878286be1, State: Initialized, Role: FOLLOWER
I20260928 02:50:17.570479 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:17.561522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35333 (local address 127.0.0.1:45675)
0928 02:50:17.561967 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:17.561985 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:17.562025 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:50:17.562915 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:17.562937 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:17.563360 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:17.563739 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:17.564073 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:17.565518 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:17.566517 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:17.567218 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:17.567402 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:17.568374 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:17.568381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:17.568400 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:17.568932 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:17.568998 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:17.569028 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:17.569129 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:17.569287 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:50:17.569496 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260928 02:50:17.570619 17969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd1cd1a558b460990ceeae878286be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260928 02:50:17.570770 17969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:17.570837 17969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:17.571071 17969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:17.572136 17969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd1cd1a558b460990ceeae878286be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260928 02:50:17.572495 17969 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd1cd1a558b460990ceeae878286be1; no voters:
I20260928 02:50:17.572913 17969 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:17.572991 17974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:17.573223 17974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd1cd1a558b460990ceeae878286be1, State: Running, Role: LEADER
I20260928 02:50:17.573544 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:17.573587 17974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd1cd1a558b460990ceeae878286be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260928 02:50:17.574266 17969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:17.574824 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd1cd1a558b460990ceeae878286be1. Latest consensus state: current_term: 1 leader_uuid: "7bd1cd1a558b460990ceeae878286be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd1cd1a558b460990ceeae878286be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260928 02:50:17.574945 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:17.575302 17983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:17.576483 17975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd1cd1a558b460990ceeae878286be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd1cd1a558b460990ceeae878286be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260928 02:50:17.576655 17975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:17.578655 17983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:17.581637 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:17.575634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33861 (local address 127.0.0.1:45675)
0928 02:50:17.575729 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:17.575733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:17.577005 (+ 1272us) server_negotiation.cc:407] Connection header received
0928 02:50:17.577174 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:17.577177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:17.577218 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:17.577266 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:17.577650 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:17.578541 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:17.579792 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:17.580037 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:17.580142 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:17.581194 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:17.581198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:17.581201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:17.581213 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:17.581241 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:17.581255 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:17.581348 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:17.581460 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:50:17.581511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:50:17.585019 17983 catalog_manager.cc:1347] Generated new cluster ID: d5e06049d98141b0b010dced5e0cce3a
I20260928 02:50:17.585090 17983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:17.592144 17983 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:17.593089 17983 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:17.597067 17983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd1cd1a558b460990ceeae878286be1: Generated new TSK 0
I20260928 02:50:17.597667 17983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:17.609850 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 17901
2026-09-28T02:50:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:17 dist-test-slave-r238 krb5kdc[18000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:17 dist-test-slave-r238 krb5kdc[18000](info): set up 2 sockets
Sep 28 02:50:17 dist-test-slave-r238 krb5kdc[18000](info): commencing operation
krb5kdc: starting...
W20260928 02:50:18.644980 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:18 dist-test-slave-r238 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:18.767627 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:18.875280 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:18.875564 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:18.875635 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:18.879732 18013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:18.879808 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:18.879848 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:18.879899 18013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:18.879937 18013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:18.885016 18013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:18.886046 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:18.887104 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:18.895491 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:18.895563 18013 server_base.cc:1034] running on GCE node
W20260928 02:50:18.895509 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:18.895488 18021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:18.896214 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:18.897347 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:18.898514 18013 hybrid_clock.cc:648] HybridClock initialized: now 1790563818898500 us; error 36 us; skew 500 ppm
Sep 28 02:50:18 dist-test-slave-r238 krb5kdc[18000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:18.905668 18013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:18.906605 18013 webserver.cc:460] Webserver started at http://127.0.0.1:34553/ using document root <none> and password file <none>
I20260928 02:50:18.907193 18013 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:18.907282 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:18.907505 18013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:18.909612 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "560d8389d2de4667ae717e33b443abe2"
format_stamp: "Formatted at 2026-09-28 02:50:18 on dist-test-slave-r238"
server_key: "7edcadb3d8000b01c988278d1950a570"
server_key_iv: "1a7ac4f41e41778268df068d7a7453f5"
server_key_version: "encryptionkey@0"
I20260928 02:50:18.910188 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "560d8389d2de4667ae717e33b443abe2"
format_stamp: "Formatted at 2026-09-28 02:50:18 on dist-test-slave-r238"
server_key: "7edcadb3d8000b01c988278d1950a570"
server_key_iv: "1a7ac4f41e41778268df068d7a7453f5"
server_key_version: "encryptionkey@0"
I20260928 02:50:18.913794 18013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:50:18.916141 18028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:18.917429 18013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:50:18.917560 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "560d8389d2de4667ae717e33b443abe2"
format_stamp: "Formatted at 2026-09-28 02:50:18 on dist-test-slave-r238"
server_key: "7edcadb3d8000b01c988278d1950a570"
server_key_iv: "1a7ac4f41e41778268df068d7a7453f5"
server_key_version: "encryptionkey@0"
I20260928 02:50:18.917675 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:18.922194 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:18.925076 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:18.925272 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:18.932155 18013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35779
I20260928 02:50:18.932168 18080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35779 every 8 connection(s)
I20260928 02:50:18.933095 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:18.935788 18081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:18.941151 18081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2: Bootstrap starting.
I20260928 02:50:18.943295 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18013
I20260928 02:50:18.943439 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:18.943703 18081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:18.943771 29666 external_mini_cluster.cc:1428] Setting key 54f68799f22a212be3a20da7337a8f5a
I20260928 02:50:18.944553 18081 log.cc:824] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:18.946831 18081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2: No bootstrap required, opened a new log
I20260928 02:50:18.949108 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:18.944970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57555 (local address 127.0.0.1:35779)
0928 02:50:18.945447 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:18.945465 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:18.945509 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:50:18.946327 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:18.946350 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:18.946693 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:18.947044 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:18.947734 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:18.947739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:18.947753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:18.947765 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:18.947824 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:18.947850 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:18.947961 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:18.948032 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:50:18.948181 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260928 02:50:18.951279 18081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560d8389d2de4667ae717e33b443abe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 02:50:18.951519 18081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:18.951592 18081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560d8389d2de4667ae717e33b443abe2, State: Initialized, Role: FOLLOWER
I20260928 02:50:18.951825 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:18.952078 18081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560d8389d2de4667ae717e33b443abe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 02:50:18.952230 18081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:18.952308 18081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:18.952404 18081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:18.953338 18081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560d8389d2de4667ae717e33b443abe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 02:50:18.953660 18081 leader_election.cc:302] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 560d8389d2de4667ae717e33b443abe2; no voters:
I20260928 02:50:18.954003 18081 leader_election.cc:288] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:18.954161 18090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:18.954334 18090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [term 1 LEADER]: Becoming Leader. State: Replica: 560d8389d2de4667ae717e33b443abe2, State: Running, Role: LEADER
I20260928 02:50:18.954674 18090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560d8389d2de4667ae717e33b443abe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } }
I20260928 02:50:18.955245 18081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:18.955734 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:18.953985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58131 (local address 127.0.0.1:35779)
0928 02:50:18.954087 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:18.954091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:18.954302 (+ 211us) server_negotiation.cc:407] Connection header received
0928 02:50:18.954477 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:18.954480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:18.954508 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:18.954560 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:18.955319 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:18.955322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:18.955324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:18.955337 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:18.955362 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:18.955374 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:18.955483 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:18.955581 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:50:18.955618 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:50:18.956291 18093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 560d8389d2de4667ae717e33b443abe2. Latest consensus state: current_term: 1 leader_uuid: "560d8389d2de4667ae717e33b443abe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560d8389d2de4667ae717e33b443abe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } } }
I20260928 02:50:18.956426 18093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:18.956807 18100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:18.957003 18092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "560d8389d2de4667ae717e33b443abe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "560d8389d2de4667ae717e33b443abe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35779 } } }
I20260928 02:50:18.957078 18092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:18.959895 18100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:18.964859 18100 catalog_manager.cc:1347] Generated new cluster ID: 6a8c3c9368cf44e9b6bb423279b6dad7
I20260928 02:50:18.964918 18100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:18.988633 18100 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:18.989017 18100 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:18.997444 18100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 560d8389d2de4667ae717e33b443abe2: Generated new TSK 0
I20260928 02:50:18.998147 18100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:19.009332 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18013
2026-09-28T02:50:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:19 dist-test-slave-r238 krb5kdc[18112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:19 dist-test-slave-r238 krb5kdc[18112](info): set up 2 sockets
Sep 28 02:50:19 dist-test-slave-r238 krb5kdc[18112](info): commencing operation
krb5kdc: starting...
W20260928 02:50:20.069065 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:20 dist-test-slave-r238 krb5kdc[18112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:20.193507 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:20.302848 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:20.303098 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:20.303138 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:20.307040 18125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:20.307091 18125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:20.307111 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:20.307127 18125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:20.307142 18125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:20.312266 18125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:20.313230 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:20.314163 18125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:20.322695 18131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:20.322645 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:20.322713 18125 server_base.cc:1034] running on GCE node
W20260928 02:50:20.322865 18130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:20.323396 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:20.324448 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:20.325613 18125 hybrid_clock.cc:648] HybridClock initialized: now 1790563820325598 us; error 37 us; skew 500 ppm
Sep 28 02:50:20 dist-test-slave-r238 krb5kdc[18112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:20.328400 18125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:20.329423 18125 webserver.cc:460] Webserver started at http://127.0.0.1:38341/ using document root <none> and password file <none>
I20260928 02:50:20.330004 18125 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:20.330063 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:20.330291 18125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:20.332283 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "8cd399cd6e9048289218092c64b50e9c"
format_stamp: "Formatted at 2026-09-28 02:50:20 on dist-test-slave-r238"
server_key: "7ce5a9a8b7f724df38af6ffae9edd98b"
server_key_iv: "ce3dca9c7aa0bd2d1f005536471daa62"
server_key_version: "encryptionkey@0"
I20260928 02:50:20.332854 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "8cd399cd6e9048289218092c64b50e9c"
format_stamp: "Formatted at 2026-09-28 02:50:20 on dist-test-slave-r238"
server_key: "7ce5a9a8b7f724df38af6ffae9edd98b"
server_key_iv: "ce3dca9c7aa0bd2d1f005536471daa62"
server_key_version: "encryptionkey@0"
I20260928 02:50:20.336462 18125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:50:20.338923 18140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:20.340209 18125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:20.340359 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "8cd399cd6e9048289218092c64b50e9c"
format_stamp: "Formatted at 2026-09-28 02:50:20 on dist-test-slave-r238"
server_key: "7ce5a9a8b7f724df38af6ffae9edd98b"
server_key_iv: "ce3dca9c7aa0bd2d1f005536471daa62"
server_key_version: "encryptionkey@0"
I20260928 02:50:20.340476 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:20.350932 18125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:20.353817 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:20.354007 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:20.360960 18125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46841
I20260928 02:50:20.360985 18192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46841 every 8 connection(s)
I20260928 02:50:20.361847 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:20.364424 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:20.369601 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18125
I20260928 02:50:20.369741 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:20.369853 18193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c: Bootstrap starting.
I20260928 02:50:20.370013 29666 external_mini_cluster.cc:1428] Setting key 56cf83829ddd0ef5128545d0c3c7f3a1
I20260928 02:50:20.372287 18193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:20.373070 18193 log.cc:824] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:20.374859 18193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c: No bootstrap required, opened a new log
I20260928 02:50:20.378494 18193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd399cd6e9048289218092c64b50e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46841 } }
Sep 28 02:50:20 dist-test-slave-r238 krb5kdc[18112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:50:20.378773 18193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:20.378831 18193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd399cd6e9048289218092c64b50e9c, State: Initialized, Role: FOLLOWER
I20260928 02:50:20.379331 18193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd399cd6e9048289218092c64b50e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46841 } }
I20260928 02:50:20.379498 18193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:20.379586 18193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:20.379710 18193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:20.380805 18193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd399cd6e9048289218092c64b50e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46841 } }
I20260928 02:50:20.381135 18193 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd399cd6e9048289218092c64b50e9c; no voters:
I20260928 02:50:20.381430 18193 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:20.381522 18198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:20.381768 18198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [term 1 LEADER]: Becoming Leader. State: Replica: 8cd399cd6e9048289218092c64b50e9c, State: Running, Role: LEADER
I20260928 02:50:20.382097 18198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd399cd6e9048289218092c64b50e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46841 } }
I20260928 02:50:20.382946 18193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:20.384418 18199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd399cd6e9048289218092c64b50e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd399cd6e9048289218092c64b50e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46841 } } }
I20260928 02:50:20.384541 18199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:20.384893 18200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd399cd6e9048289218092c64b50e9c. Latest consensus state: current_term: 1 leader_uuid: "8cd399cd6e9048289218092c64b50e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd399cd6e9048289218092c64b50e9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46841 } } }
I20260928 02:50:20.384989 18200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:20.385381 18203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:20.385448 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:20.371323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:51817 (local address 127.0.0.1:46841)
0928 02:50:20.371804 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:20.371820 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:20.371855 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:20.372537 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:20.372555 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:20.372978 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:20.373305 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:20.374117 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:20.375129 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:20.375974 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:20.376445 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:20.376642 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:20.379435 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:20.379440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:20.379452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:20.379501 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:20.381496 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:20.382115 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:20.382121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:20.382126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:20.382268 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:20.382757 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:20.382760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:20.382762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:20.383308 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:20.383599 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:20.384119 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 02:50:20.384367 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260928 02:50:20.388213 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:20.389364 18203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:20.395869 18203 catalog_manager.cc:1347] Generated new cluster ID: a458ccffa70d416d9ecefd325a5d3f71
I20260928 02:50:20.395953 18203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:20.400950 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:20.391887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:40887 (local address 127.0.0.1:46841)
0928 02:50:20.391988 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:20.391992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:20.392295 (+ 303us) server_negotiation.cc:407] Connection header received
0928 02:50:20.392429 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:20.392433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:20.392480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:20.392566 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:20.393417 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:20.394092 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:20.394898 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:20.395136 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:20.395253 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:20.397043 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:20.397046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:20.397049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:20.397074 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:20.398816 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:20.399370 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:20.399374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:20.399376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:20.399473 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:20.399819 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:20.399822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:20.399824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:20.400035 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:20.400187 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:20.400461 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:50:20.400790 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:50:20.411504 18203 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:20.412518 18203 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:20.419118 18203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd399cd6e9048289218092c64b50e9c: Generated new TSK 0
I20260928 02:50:20.419868 18203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:20.429133 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18125
W20260928 02:50:20.434839 18110 connection.cc:535] client connection to 127.0.0.1:46841 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46841): Connection reset by peer (error 104)
2026-09-28T02:50:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:20 dist-test-slave-r238 krb5kdc[18224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:20 dist-test-slave-r238 krb5kdc[18224](info): set up 2 sockets
Sep 28 02:50:20 dist-test-slave-r238 krb5kdc[18224](info): commencing operation
krb5kdc: starting...
W20260928 02:50:21.560935 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:21 dist-test-slave-r238 krb5kdc[18224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:21.680330 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:21.790470 18237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:21.790732 18237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:21.790773 18237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:21.794850 18237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:21.794910 18237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:21.794930 18237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:21.794947 18237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:21.794961 18237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:21.800139 18237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:21.801143 18237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:21.802150 18237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:21.810969 18243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:21.811000 18245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:21.811002 18242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:21.811053 18237 server_base.cc:1034] running on GCE node
I20260928 02:50:21.811877 18237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:21.813028 18237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:21.814184 18237 hybrid_clock.cc:648] HybridClock initialized: now 1790563821814157 us; error 41 us; skew 500 ppm
Sep 28 02:50:21 dist-test-slave-r238 krb5kdc[18224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:21.816898 18237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:21.817731 18237 webserver.cc:460] Webserver started at http://127.0.0.1:34939/ using document root <none> and password file <none>
I20260928 02:50:21.818228 18237 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:21.818284 18237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:21.818439 18237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:21.820418 18237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "a12290a7e4db4d918e10bb26ad801ff4"
format_stamp: "Formatted at 2026-09-28 02:50:21 on dist-test-slave-r238"
server_key: "35e49272d3d8f16d0513f5f1e8d11f11"
server_key_iv: "42cbd25ad8290692ed4f6689660869d7"
server_key_version: "encryptionkey@0"
I20260928 02:50:21.820988 18237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "a12290a7e4db4d918e10bb26ad801ff4"
format_stamp: "Formatted at 2026-09-28 02:50:21 on dist-test-slave-r238"
server_key: "35e49272d3d8f16d0513f5f1e8d11f11"
server_key_iv: "42cbd25ad8290692ed4f6689660869d7"
server_key_version: "encryptionkey@0"
I20260928 02:50:21.824440 18237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:21.826742 18252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:21.827965 18237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:21.828110 18237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "a12290a7e4db4d918e10bb26ad801ff4"
format_stamp: "Formatted at 2026-09-28 02:50:21 on dist-test-slave-r238"
server_key: "35e49272d3d8f16d0513f5f1e8d11f11"
server_key_iv: "42cbd25ad8290692ed4f6689660869d7"
server_key_version: "encryptionkey@0"
I20260928 02:50:21.828219 18237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:21.840247 18237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:21.843209 18237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:21.843427 18237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:21.850739 18237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40793
I20260928 02:50:21.850760 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40793 every 8 connection(s)
I20260928 02:50:21.851686 18237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:21.854331 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:21.856417 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18237
I20260928 02:50:21.856562 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:21.856923 29666 external_mini_cluster.cc:1428] Setting key 1fceb858f9f2db472f39dfdbc2fb353b
I20260928 02:50:21.861474 18305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4: Bootstrap starting.
I20260928 02:50:21.863624 18305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:21.864467 18305 log.cc:824] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:21.866307 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:21.858311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:35749 (local address 127.0.0.1:40793)
0928 02:50:21.858724 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:21.858739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:21.858772 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:50:21.859630 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:21.859654 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:21.860128 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:21.860532 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:21.860962 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:21.861980 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:21.862874 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:21.863349 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:21.863516 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:21.864411 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:21.864416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:21.864428 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:21.864943 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:21.865007 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:21.865040 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:21.865186 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:21.865435 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:50:21.865596 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260928 02:50:21.866559 18305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4: No bootstrap required, opened a new log
I20260928 02:50:21.869256 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:21.870935 18305 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12290a7e4db4d918e10bb26ad801ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260928 02:50:21.871198 18305 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:21.871281 18305 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12290a7e4db4d918e10bb26ad801ff4, State: Initialized, Role: FOLLOWER
I20260928 02:50:21.871786 18305 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12290a7e4db4d918e10bb26ad801ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260928 02:50:21.871968 18305 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:21.872044 18305 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:21.872146 18305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:21.873306 18305 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12290a7e4db4d918e10bb26ad801ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260928 02:50:21.873665 18305 leader_election.cc:302] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12290a7e4db4d918e10bb26ad801ff4; no voters:
I20260928 02:50:21.873945 18305 leader_election.cc:288] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:21.874075 18315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:21.874312 18315 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [term 1 LEADER]: Becoming Leader. State: Replica: a12290a7e4db4d918e10bb26ad801ff4, State: Running, Role: LEADER
I20260928 02:50:21.874680 18315 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12290a7e4db4d918e10bb26ad801ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } }
I20260928 02:50:21.875115 18305 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:21.876354 18316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12290a7e4db4d918e10bb26ad801ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12290a7e4db4d918e10bb26ad801ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } } }
I20260928 02:50:21.876482 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:21.871532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:60105 (local address 127.0.0.1:40793)
0928 02:50:21.871650 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:21.871653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:21.871905 (+ 252us) server_negotiation.cc:407] Connection header received
0928 02:50:21.872025 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:21.872028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:21.872079 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:21.872162 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:21.872464 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:21.873248 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:21.874283 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:21.874628 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:21.874709 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:21.875747 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:21.875751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:21.875753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:21.875766 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:21.875792 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:21.875803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:21.875886 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:21.876251 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 02:50:21.876293 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:21.876494 18316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:21.877031 18317 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12290a7e4db4d918e10bb26ad801ff4. Latest consensus state: current_term: 1 leader_uuid: "a12290a7e4db4d918e10bb26ad801ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12290a7e4db4d918e10bb26ad801ff4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40793 } } }
I20260928 02:50:21.877153 18317 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:21.877529 18328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:21.880434 18328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:21.885514 18328 catalog_manager.cc:1347] Generated new cluster ID: ad5ed1b7f56644db879fb14ad95e1f20
I20260928 02:50:21.885591 18328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:21.928371 18328 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:21.929489 18328 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:21.936524 18328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12290a7e4db4d918e10bb26ad801ff4: Generated new TSK 0
I20260928 02:50:21.937309 18328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:21.942909 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18237
W20260928 02:50:21.948176 18222 connection.cc:535] client connection to 127.0.0.1:40793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40793): Connection reset by peer (error 104)
2026-09-28T02:50:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:21 dist-test-slave-r238 krb5kdc[18336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:21 dist-test-slave-r238 krb5kdc[18336](info): set up 2 sockets
Sep 28 02:50:21 dist-test-slave-r238 krb5kdc[18336](info): commencing operation
krb5kdc: starting...
W20260928 02:50:22.981153 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:23 dist-test-slave-r238 krb5kdc[18336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:23.107810 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:23.216063 18349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:23.216336 18349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:23.216377 18349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:23.220350 18349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:23.220415 18349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:23.220435 18349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:23.220451 18349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:23.220466 18349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:23.225757 18349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:23.226787 18349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:23.227751 18349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:23.236513 18355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:23.236536 18354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:23.236729 18357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:23.236758 18349 server_base.cc:1034] running on GCE node
I20260928 02:50:23.237349 18349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:23.238471 18349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:23.239630 18349 hybrid_clock.cc:648] HybridClock initialized: now 1790563823239616 us; error 27 us; skew 500 ppm
Sep 28 02:50:23 dist-test-slave-r238 krb5kdc[18336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:23.242511 18349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:23.243492 18349 webserver.cc:460] Webserver started at http://127.0.0.1:38255/ using document root <none> and password file <none>
I20260928 02:50:23.244042 18349 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:23.244133 18349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:23.244362 18349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:23.246435 18349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "834e38b1ca734e849e234cd51e39fc10"
format_stamp: "Formatted at 2026-09-28 02:50:23 on dist-test-slave-r238"
server_key: "45849304a038c796ff5ac8be62704343"
server_key_iv: "1196df7fb21ed6ba4ce2f4fbedd7be51"
server_key_version: "encryptionkey@0"
I20260928 02:50:23.246997 18349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "834e38b1ca734e849e234cd51e39fc10"
format_stamp: "Formatted at 2026-09-28 02:50:23 on dist-test-slave-r238"
server_key: "45849304a038c796ff5ac8be62704343"
server_key_iv: "1196df7fb21ed6ba4ce2f4fbedd7be51"
server_key_version: "encryptionkey@0"
I20260928 02:50:23.250662 18349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:50:23.253015 18364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:23.254261 18349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:23.254428 18349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "834e38b1ca734e849e234cd51e39fc10"
format_stamp: "Formatted at 2026-09-28 02:50:23 on dist-test-slave-r238"
server_key: "45849304a038c796ff5ac8be62704343"
server_key_iv: "1196df7fb21ed6ba4ce2f4fbedd7be51"
server_key_version: "encryptionkey@0"
I20260928 02:50:23.254544 18349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:23.259008 18349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:23.262244 18349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:23.262434 18349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:23.269557 18349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33167
I20260928 02:50:23.269582 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33167 every 8 connection(s)
I20260928 02:50:23.270437 18349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:23.273190 18417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:23.273581 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18349
I20260928 02:50:23.273710 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:23.273976 29666 external_mini_cluster.cc:1428] Setting key 6faeb92e8a12edbcd570e294485a6969
I20260928 02:50:23.279491 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:23.275300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:50349 (local address 127.0.0.1:33167)
0928 02:50:23.275714 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:23.275734 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:23.275779 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:50:23.276574 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:23.276599 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:23.277039 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:23.277479 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:23.278131 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:23.278137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:23.278150 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:23.278161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:23.278213 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:23.278238 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:23.278373 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:23.278439 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:50:23.278587 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260928 02:50:23.280789 18417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10: Bootstrap starting.
I20260928 02:50:23.282415 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:23.283102 18417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:23.283831 18417 log.cc:824] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:23.285651 18417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10: No bootstrap required, opened a new log
I20260928 02:50:23.287256 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:23.284445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:52711 (local address 127.0.0.1:33167)
0928 02:50:23.284571 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:23.284575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:23.285979 (+ 1404us) server_negotiation.cc:407] Connection header received
0928 02:50:23.286107 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:23.286109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:23.286137 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:23.286197 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:23.286856 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:23.286859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:23.286860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:23.286870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:23.286894 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:23.286906 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:23.286973 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:23.287078 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 02:50:23.287110 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:50:23.290045 18417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834e38b1ca734e849e234cd51e39fc10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33167 } }
I20260928 02:50:23.290278 18417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:23.290344 18417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834e38b1ca734e849e234cd51e39fc10, State: Initialized, Role: FOLLOWER
I20260928 02:50:23.290766 18417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834e38b1ca734e849e234cd51e39fc10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33167 } }
I20260928 02:50:23.290905 18417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:23.290980 18417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:23.291128 18417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:23.291975 18417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834e38b1ca734e849e234cd51e39fc10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33167 } }
I20260928 02:50:23.292356 18417 leader_election.cc:302] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 834e38b1ca734e849e234cd51e39fc10; no voters:
I20260928 02:50:23.292688 18417 leader_election.cc:288] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:23.292752 18427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:23.293001 18427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [term 1 LEADER]: Becoming Leader. State: Replica: 834e38b1ca734e849e234cd51e39fc10, State: Running, Role: LEADER
I20260928 02:50:23.293322 18427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834e38b1ca734e849e234cd51e39fc10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33167 } }
I20260928 02:50:23.293943 18417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:23.294971 18429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834e38b1ca734e849e234cd51e39fc10. Latest consensus state: current_term: 1 leader_uuid: "834e38b1ca734e849e234cd51e39fc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834e38b1ca734e849e234cd51e39fc10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33167 } } }
I20260928 02:50:23.295090 18429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:23.295485 18436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:23.295475 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "834e38b1ca734e849e234cd51e39fc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834e38b1ca734e849e234cd51e39fc10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33167 } } }
I20260928 02:50:23.295697 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:23.298612 18436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:23.303997 18436 catalog_manager.cc:1347] Generated new cluster ID: d808ac6f86f44ea6b351310811313865
I20260928 02:50:23.304075 18436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:23.325884 18436 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:23.326260 18436 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:23.336143 18436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 834e38b1ca734e849e234cd51e39fc10: Generated new TSK 0
I20260928 02:50:23.336894 18436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:23.341117 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18349
2026-09-28T02:50:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:23 dist-test-slave-r238 krb5kdc[18448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:23 dist-test-slave-r238 krb5kdc[18448](info): set up 2 sockets
Sep 28 02:50:23 dist-test-slave-r238 krb5kdc[18448](info): commencing operation
krb5kdc: starting...
W20260928 02:50:24.372859 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:24 dist-test-slave-r238 krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:24.496173 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:24.603803 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:24.604100 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:24.604175 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:24.608094 18461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:24.608189 18461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:24.608228 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:24.608287 18461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:24.608323 18461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:24.613458 18461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:24.614490 18461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:24.615571 18461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:24.623836 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:24.623881 18466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:24.623896 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:24.624014 18461 server_base.cc:1034] running on GCE node
I20260928 02:50:24.624595 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:24.625746 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:24.626931 18461 hybrid_clock.cc:648] HybridClock initialized: now 1790563824626904 us; error 40 us; skew 500 ppm
Sep 28 02:50:24 dist-test-slave-r238 krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:24.634595 18461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:24.635538 18461 webserver.cc:460] Webserver started at http://127.0.0.1:44223/ using document root <none> and password file <none>
I20260928 02:50:24.636062 18461 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:24.636121 18461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:24.636281 18461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:24.638319 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "e9f7b97c1518492882d8b42632d59f7b"
format_stamp: "Formatted at 2026-09-28 02:50:24 on dist-test-slave-r238"
server_key: "237bffb49a2619ab930e464e69220334"
server_key_iv: "e06d4ca6db7c25e20fdaa2eccccbbfd0"
server_key_version: "encryptionkey@0"
I20260928 02:50:24.638825 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "e9f7b97c1518492882d8b42632d59f7b"
format_stamp: "Formatted at 2026-09-28 02:50:24 on dist-test-slave-r238"
server_key: "237bffb49a2619ab930e464e69220334"
server_key_iv: "e06d4ca6db7c25e20fdaa2eccccbbfd0"
server_key_version: "encryptionkey@0"
I20260928 02:50:24.642457 18461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:50:24.644886 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:24.646322 18461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:50:24.646554 18461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "e9f7b97c1518492882d8b42632d59f7b"
format_stamp: "Formatted at 2026-09-28 02:50:24 on dist-test-slave-r238"
server_key: "237bffb49a2619ab930e464e69220334"
server_key_iv: "e06d4ca6db7c25e20fdaa2eccccbbfd0"
server_key_version: "encryptionkey@0"
I20260928 02:50:24.646669 18461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:24.666782 18461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:24.669963 18461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:24.670173 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:24.677388 18461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41565
I20260928 02:50:24.677404 18528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41565 every 8 connection(s)
I20260928 02:50:24.678346 18461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:24.681252 18529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:24.682150 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18461
I20260928 02:50:24.682291 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:24.682570 29666 external_mini_cluster.cc:1428] Setting key 0951d59eb00c3381b9246c644308291e
I20260928 02:50:24.687932 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b: Bootstrap starting.
I20260928 02:50:24.690582 18529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:24.691305 18529 log.cc:824] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:50:24 dist-test-slave-r238 krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:50:24.693662 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b: No bootstrap required, opened a new log
I20260928 02:50:24.698253 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:24.683963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:40493 (local address 127.0.0.1:41565)
0928 02:50:24.684372 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:24.684391 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:24.684435 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:50:24.685287 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:24.685310 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:24.685735 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:24.686143 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:24.686848 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:24.688293 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:24.689227 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:24.689914 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:24.690073 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:24.692954 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:24.692960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:24.692975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:24.693007 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:24.694992 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:24.695482 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:24.695488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:24.695494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:24.695631 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:24.695945 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:24.695950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:24.695952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:24.696547 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:24.696806 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:24.697015 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:50:24.697249 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20260928 02:50:24.698467 18529 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b97c1518492882d8b42632d59f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260928 02:50:24.698743 18529 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:24.698804 18529 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f7b97c1518492882d8b42632d59f7b, State: Initialized, Role: FOLLOWER
I20260928 02:50:24.699226 18529 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b97c1518492882d8b42632d59f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260928 02:50:24.699388 18529 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:24.699458 18529 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:24.699558 18529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:24.700596 18529 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b97c1518492882d8b42632d59f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260928 02:50:24.700968 18529 leader_election.cc:302] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f7b97c1518492882d8b42632d59f7b; no voters:
I20260928 02:50:24.701313 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:24.701314 18529 leader_election.cc:288] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:24.701448 18534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:24.701680 18534 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [term 1 LEADER]: Becoming Leader. State: Replica: e9f7b97c1518492882d8b42632d59f7b, State: Running, Role: LEADER
I20260928 02:50:24.702026 18534 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b97c1518492882d8b42632d59f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260928 02:50:24.702649 18529 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:24.703636 18535 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f7b97c1518492882d8b42632d59f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b97c1518492882d8b42632d59f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } } }
I20260928 02:50:24.703788 18535 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:24.704111 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f7b97c1518492882d8b42632d59f7b. Latest consensus state: current_term: 1 leader_uuid: "e9f7b97c1518492882d8b42632d59f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f7b97c1518492882d8b42632d59f7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } } }
I20260928 02:50:24.704183 18548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:24.704200 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:24.707298 18548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:24.713068 18548 catalog_manager.cc:1347] Generated new cluster ID: a2350d11d161495e8c961ef7c16edb3b
I20260928 02:50:24.713126 18548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:24.713521 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:24.703773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:55227 (local address 127.0.0.1:41565)
0928 02:50:24.703875 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:24.703880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:24.703896 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:50:24.703935 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:24.703938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:24.703981 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:24.704056 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:24.705084 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:24.705915 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:24.707087 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:24.707745 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:24.707883 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:24.709737 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:24.709741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:24.709744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:24.709783 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:24.711524 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:24.712070 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:24.712074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:24.712076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:24.712163 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:24.712538 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:24.712542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:24.712543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:24.712787 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:24.712936 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:24.713313 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 02:50:24.713403 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:50:24.726843 18548 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:24.727715 18548 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:24.737548 18548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f7b97c1518492882d8b42632d59f7b: Generated new TSK 0
I20260928 02:50:24.738262 18548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:24.743007 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18461
W20260928 02:50:24.747802 18446 connection.cc:535] client connection to 127.0.0.1:41565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41565): Connection reset by peer (error 104)
2026-09-28T02:50:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:24 dist-test-slave-r238 krb5kdc[18560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:24 dist-test-slave-r238 krb5kdc[18560](info): set up 2 sockets
Sep 28 02:50:24 dist-test-slave-r238 krb5kdc[18560](info): commencing operation
krb5kdc: starting...
W20260928 02:50:25.780995 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:25 dist-test-slave-r238 krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:25.910915 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:26.020057 18573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:26.020327 18573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:26.020370 18573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:26.024436 18573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:26.024498 18573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:26.024518 18573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:26.024534 18573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:26.024549 18573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:26.029789 18573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:26.030731 18573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:26.031684 18573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:26.040105 18579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:26.040107 18581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:26.040105 18578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:26.040355 18573 server_base.cc:1034] running on GCE node
I20260928 02:50:26.040910 18573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:26.041988 18573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:26.043149 18573 hybrid_clock.cc:648] HybridClock initialized: now 1790563826043134 us; error 28 us; skew 500 ppm
Sep 28 02:50:26 dist-test-slave-r238 krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:26.045954 18573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:26.046787 18573 webserver.cc:460] Webserver started at http://127.0.0.1:46843/ using document root <none> and password file <none>
I20260928 02:50:26.047338 18573 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:26.047397 18573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:26.047611 18573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:26.049664 18573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "70e75ff26872420685d5644a87db410e"
format_stamp: "Formatted at 2026-09-28 02:50:26 on dist-test-slave-r238"
server_key: "5e0a1a288edb874bd0d8227b91a70b89"
server_key_iv: "3bcff77b721814e2b5f7cd6c8e450456"
server_key_version: "encryptionkey@0"
I20260928 02:50:26.050225 18573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "70e75ff26872420685d5644a87db410e"
format_stamp: "Formatted at 2026-09-28 02:50:26 on dist-test-slave-r238"
server_key: "5e0a1a288edb874bd0d8227b91a70b89"
server_key_iv: "3bcff77b721814e2b5f7cd6c8e450456"
server_key_version: "encryptionkey@0"
I20260928 02:50:26.053844 18573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:50:26.056169 18588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:26.057519 18573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:26.057654 18573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "70e75ff26872420685d5644a87db410e"
format_stamp: "Formatted at 2026-09-28 02:50:26 on dist-test-slave-r238"
server_key: "5e0a1a288edb874bd0d8227b91a70b89"
server_key_iv: "3bcff77b721814e2b5f7cd6c8e450456"
server_key_version: "encryptionkey@0"
I20260928 02:50:26.057765 18573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:26.082351 18573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:26.085525 18573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:26.085733 18573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:26.092934 18573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40081
I20260928 02:50:26.092960 18640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40081 every 8 connection(s)
I20260928 02:50:26.093890 18573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:26.096652 18641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:26.097254 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18573
I20260928 02:50:26.097388 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:26.097667 29666 external_mini_cluster.cc:1428] Setting key 74203002a4f1ad61faf20851bb8d21a3
I20260928 02:50:26.102619 18641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e: Bootstrap starting.
I20260928 02:50:26.104928 18641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:26.105616 18641 log.cc:824] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:26.106540 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:26.099095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:40553 (local address 127.0.0.1:40081)
0928 02:50:26.099529 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:26.099566 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:26.099603 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:50:26.100232 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:26.100248 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:26.100589 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:26.101000 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:26.101385 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:26.102333 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:26.103263 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:26.103738 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:26.103918 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:26.104879 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:26.104884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:26.104896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:26.105334 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:26.105386 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:26.105410 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:26.105546 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:26.105727 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 02:50:26.105859 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":137,"threads_started":1}
I20260928 02:50:26.107497 18641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e: No bootstrap required, opened a new log
I20260928 02:50:26.109273 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:26.111055 18641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e75ff26872420685d5644a87db410e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 02:50:26.111384 18641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:26.111452 18641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e75ff26872420685d5644a87db410e, State: Initialized, Role: FOLLOWER
I20260928 02:50:26.111956 18641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e75ff26872420685d5644a87db410e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 02:50:26.112114 18641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:26.112191 18641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:26.112316 18641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:26.113227 18641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e75ff26872420685d5644a87db410e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 02:50:26.113555 18641 leader_election.cc:302] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e75ff26872420685d5644a87db410e; no voters:
I20260928 02:50:26.113874 18641 leader_election.cc:288] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:26.113940 18651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:26.114132 18651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [term 1 LEADER]: Becoming Leader. State: Replica: 70e75ff26872420685d5644a87db410e, State: Running, Role: LEADER
I20260928 02:50:26.114415 18651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e75ff26872420685d5644a87db410e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } }
I20260928 02:50:26.115091 18641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:26.115497 18651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e75ff26872420685d5644a87db410e. Latest consensus state: current_term: 1 leader_uuid: "70e75ff26872420685d5644a87db410e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e75ff26872420685d5644a87db410e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } } }
I20260928 02:50:26.115597 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:26.111217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:36403 (local address 127.0.0.1:40081)
0928 02:50:26.111342 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:26.111346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:26.111551 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:50:26.111661 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:26.111663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:26.111698 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:26.111785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:26.112054 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:26.112778 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:26.113656 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:26.114013 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:26.114107 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:26.115025 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:26.115028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:26.115031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:26.115043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:26.115072 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:26.115083 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:26.115174 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:26.115367 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:50:26.115409 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:26.115628 18651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:26.116094 18651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e75ff26872420685d5644a87db410e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e75ff26872420685d5644a87db410e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40081 } } }
I20260928 02:50:26.116184 18651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:26.116482 18659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:26.119565 18659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:26.125363 18659 catalog_manager.cc:1347] Generated new cluster ID: 4d5d0e1d99484e1982f47223966bfab5
I20260928 02:50:26.125437 18659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:26.142947 18659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:26.143888 18659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:26.149955 18659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e75ff26872420685d5644a87db410e: Generated new TSK 0
I20260928 02:50:26.150655 18659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:26.154462 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18573
W20260928 02:50:26.159562 18558 connection.cc:535] client connection to 127.0.0.1:40081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40081): Connection reset by peer (error 104)
2026-09-28T02:50:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:26 dist-test-slave-r238 krb5kdc[18672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:26 dist-test-slave-r238 krb5kdc[18672](info): set up 2 sockets
Sep 28 02:50:26 dist-test-slave-r238 krb5kdc[18672](info): commencing operation
krb5kdc: starting...
W20260928 02:50:27.188930 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:27 dist-test-slave-r238 krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:50:27.312692 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:27.419378 18685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:27.419648 18685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:27.419690 18685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:27.423774 18685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:27.423838 18685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:27.423858 18685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:27.423874 18685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:27.423890 18685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:27.429651 18685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:27.430814 18685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:27.431933 18685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:27.440701 18693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:27.440701 18691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:27.440713 18690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:27.441035 18685 server_base.cc:1034] running on GCE node
I20260928 02:50:27.441642 18685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:27.442780 18685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:27.443969 18685 hybrid_clock.cc:648] HybridClock initialized: now 1790563827443956 us; error 27 us; skew 500 ppm
Sep 28 02:50:27 dist-test-slave-r238 krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:27.446770 18685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:50:27.447737 18685 webserver.cc:460] Webserver started at http://127.0.0.1:41637/ using document root <none> and password file <none>
I20260928 02:50:27.448257 18685 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:27.448315 18685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:27.448485 18685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:27.450575 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "81a9111fe8b848dba7cf4828e867bddc"
format_stamp: "Formatted at 2026-09-28 02:50:27 on dist-test-slave-r238"
server_key: "56e3905a279d4b992644e9d57f1b5780"
server_key_iv: "9a0a9c145500fc8326b39d2799f64bb9"
server_key_version: "encryptionkey@0"
I20260928 02:50:27.451097 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "81a9111fe8b848dba7cf4828e867bddc"
format_stamp: "Formatted at 2026-09-28 02:50:27 on dist-test-slave-r238"
server_key: "56e3905a279d4b992644e9d57f1b5780"
server_key_iv: "9a0a9c145500fc8326b39d2799f64bb9"
server_key_version: "encryptionkey@0"
I20260928 02:50:27.454838 18685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:50:27.457248 18700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:27.458518 18685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:27.458648 18685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "81a9111fe8b848dba7cf4828e867bddc"
format_stamp: "Formatted at 2026-09-28 02:50:27 on dist-test-slave-r238"
server_key: "56e3905a279d4b992644e9d57f1b5780"
server_key_iv: "9a0a9c145500fc8326b39d2799f64bb9"
server_key_version: "encryptionkey@0"
I20260928 02:50:27.458750 18685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:27.463266 18685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:27.466639 18685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:27.466817 18685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:27.473610 18685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36661
I20260928 02:50:27.473677 18752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36661 every 8 connection(s)
I20260928 02:50:27.474597 18685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:27.477340 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:27.478220 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18685
I20260928 02:50:27.478350 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:27.478617 29666 external_mini_cluster.cc:1428] Setting key 7cc9ba700db761b30c6ec3ff55317daa
I20260928 02:50:27.483476 18753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc: Bootstrap starting.
I20260928 02:50:27.483825 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:27.479987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:36059 (local address 127.0.0.1:36661)
0928 02:50:27.480425 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:27.480439 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:27.480474 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:27.481167 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:27.481185 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:27.481477 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:27.481812 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:27.482624 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:27.482628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:27.482640 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:27.482650 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:27.482701 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:27.482720 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:27.482869 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:27.482942 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:50:27.483106 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260928 02:50:27.486013 18753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:27.486622 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:27.486837 18753 log.cc:824] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:27.489171 18753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc: No bootstrap required, opened a new log
I20260928 02:50:27.490547 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:27.488826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.101:57775 (local address 127.0.0.1:36661)
0928 02:50:27.488953 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:27.488957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:27.489178 (+ 221us) server_negotiation.cc:407] Connection header received
0928 02:50:27.489348 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:27.489350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:27.489379 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:27.489453 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:27.490198 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:27.490201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:27.490204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:27.490214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:27.490237 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:27.490247 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:27.490310 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:27.490425 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:50:27.490459 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:50:27.493379 18753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9111fe8b848dba7cf4828e867bddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260928 02:50:27.493597 18753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:27.493676 18753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a9111fe8b848dba7cf4828e867bddc, State: Initialized, Role: FOLLOWER
I20260928 02:50:27.494122 18753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9111fe8b848dba7cf4828e867bddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260928 02:50:27.494266 18753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:27.494355 18753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:27.494472 18753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:27.495581 18753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9111fe8b848dba7cf4828e867bddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260928 02:50:27.495923 18753 leader_election.cc:302] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81a9111fe8b848dba7cf4828e867bddc; no voters:
I20260928 02:50:27.496214 18753 leader_election.cc:288] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:27.496340 18763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:27.496578 18763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [term 1 LEADER]: Becoming Leader. State: Replica: 81a9111fe8b848dba7cf4828e867bddc, State: Running, Role: LEADER
I20260928 02:50:27.496963 18763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9111fe8b848dba7cf4828e867bddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } }
I20260928 02:50:27.497524 18753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:27.498749 18764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81a9111fe8b848dba7cf4828e867bddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9111fe8b848dba7cf4828e867bddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } } }
I20260928 02:50:27.498752 18765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81a9111fe8b848dba7cf4828e867bddc. Latest consensus state: current_term: 1 leader_uuid: "81a9111fe8b848dba7cf4828e867bddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a9111fe8b848dba7cf4828e867bddc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36661 } } }
I20260928 02:50:27.498935 18764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:27.498987 18765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc [sys.catalog]: This master's current role is: LEADER
W20260928 02:50:27.502281 18778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:27.502369 18778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:27.502452 18774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:27.503139 18774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:27.508291 18774 catalog_manager.cc:1347] Generated new cluster ID: 6201e5899bb24456ad312c52dcf65555
I20260928 02:50:27.508351 18774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:27.520999 18774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:27.521332 18774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:27.528296 18774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81a9111fe8b848dba7cf4828e867bddc: Generated new TSK 0
I20260928 02:50:27.529121 18774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:27.534422 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18685
2026-09-28T02:50:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:27 dist-test-slave-r238 krb5kdc[18784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:27 dist-test-slave-r238 krb5kdc[18784](info): set up 2 sockets
Sep 28 02:50:27 dist-test-slave-r238 krb5kdc[18784](info): commencing operation
krb5kdc: starting...
W20260928 02:50:28.588922 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:28 dist-test-slave-r238 krb5kdc[18784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:28.709792 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43031
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:43031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:28.819854 18797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:28.820120 18797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:28.820161 18797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:28.824085 18797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:28.824143 18797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:28.824163 18797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:28.824179 18797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:28.824194 18797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:28.829315 18797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:43031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:28.830296 18797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:28.831274 18797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:28.839661 18805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:28.839684 18803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:28.839722 18802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:28.839766 18797 server_base.cc:1034] running on GCE node
I20260928 02:50:28.840519 18797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:28.841611 18797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:28.842765 18797 hybrid_clock.cc:648] HybridClock initialized: now 1790563828842738 us; error 42 us; skew 500 ppm
Sep 28 02:50:28 dist-test-slave-r238 krb5kdc[18784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563828, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:28.845549 18797 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:28.846454 18797 webserver.cc:460] Webserver started at http://127.28.248.190:38009/ using document root <none> and password file <none>
I20260928 02:50:28.847024 18797 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:28.847117 18797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:28.847350 18797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:28.849440 18797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "1ec27f90afab461280b7d5d02e4ebef5"
format_stamp: "Formatted at 2026-09-28 02:50:28 on dist-test-slave-r238"
server_key: "aa73f9442a27fb4d8311cf93ae123a22"
server_key_iv: "22478b50791a965745360196fddfa463"
server_key_version: "encryptionkey@0"
I20260928 02:50:28.850008 18797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "1ec27f90afab461280b7d5d02e4ebef5"
format_stamp: "Formatted at 2026-09-28 02:50:28 on dist-test-slave-r238"
server_key: "aa73f9442a27fb4d8311cf93ae123a22"
server_key_iv: "22478b50791a965745360196fddfa463"
server_key_version: "encryptionkey@0"
I20260928 02:50:28.853816 18797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:50:28.856181 18812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:28.857494 18797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:50:28.857635 18797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "1ec27f90afab461280b7d5d02e4ebef5"
format_stamp: "Formatted at 2026-09-28 02:50:28 on dist-test-slave-r238"
server_key: "aa73f9442a27fb4d8311cf93ae123a22"
server_key_iv: "22478b50791a965745360196fddfa463"
server_key_version: "encryptionkey@0"
I20260928 02:50:28.857744 18797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:28.875473 18797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:28.878497 18797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:28.878695 18797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:28.885773 18797 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:43031
I20260928 02:50:28.885803 18864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:43031 every 8 connection(s)
I20260928 02:50:28.886713 18797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:28.889407 18865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:28.894692 18865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5: Bootstrap starting.
I20260928 02:50:28.896076 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18797
I20260928 02:50:28.896220 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:28.896504 29666 external_mini_cluster.cc:1428] Setting key 8059d36e000dd167a93be5b984381008
I20260928 02:50:28.897430 18865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:28.898262 18865 log.cc:824] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:28.900069 18865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5: No bootstrap required, opened a new log
I20260928 02:50:28.903477 18865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec27f90afab461280b7d5d02e4ebef5" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43031 } }
I20260928 02:50:28.903759 18865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:28.903833 18865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec27f90afab461280b7d5d02e4ebef5, State: Initialized, Role: FOLLOWER
I20260928 02:50:28.904289 18865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec27f90afab461280b7d5d02e4ebef5" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43031 } }
I20260928 02:50:28.904433 18865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:28.904522 18865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:28.904642 18865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 02:50:28 dist-test-slave-r238 krb5kdc[18784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:28.905627 18865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec27f90afab461280b7d5d02e4ebef5" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43031 } }
I20260928 02:50:28.905951 18865 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec27f90afab461280b7d5d02e4ebef5; no voters:
I20260928 02:50:28.906248 18865 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:28.906589 18870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:28.906805 18870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec27f90afab461280b7d5d02e4ebef5, State: Running, Role: LEADER
I20260928 02:50:28.907119 18870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec27f90afab461280b7d5d02e4ebef5" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43031 } }
I20260928 02:50:28.907521 18865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:28.908716 18871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec27f90afab461280b7d5d02e4ebef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec27f90afab461280b7d5d02e4ebef5" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43031 } } }
I20260928 02:50:28.908842 18871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:28.909211 18872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec27f90afab461280b7d5d02e4ebef5. Latest consensus state: current_term: 1 leader_uuid: "1ec27f90afab461280b7d5d02e4ebef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec27f90afab461280b7d5d02e4ebef5" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 43031 } } }
I20260928 02:50:28.909323 18872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5 [sys.catalog]: This master's current role is: LEADER
W20260928 02:50:28.912214 18885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:28.912307 18885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:28.912415 18880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:28.913332 18880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:28.914162 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:28.897639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53393 (local address 127.28.248.190:43031)
0928 02:50:28.898070 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:28.898084 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:28.898119 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:28.898818 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:28.898837 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:28.899176 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:28.899510 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:28.900382 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:28.901412 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:28.902273 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:28.902754 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:28.902931 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:28.905909 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:28.905920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:28.905934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:28.905960 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:28.910747 (+ 4787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:28.911352 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:28.911359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:28.911365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:28.911513 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:28.911965 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:28.911969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:28.911970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:28.912451 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:28.912698 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:28.912995 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:50:28.913226 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260928 02:50:28.917768 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:28.919150 18880 catalog_manager.cc:1347] Generated new cluster ID: f19a9f4364584f29a489ae8335fd46d9
I20260928 02:50:28.919214 18880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:28.932426 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:28.921080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52847 (local address 127.28.248.190:43031)
0928 02:50:28.921189 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:28.921193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:28.924902 (+ 3709us) server_negotiation.cc:407] Connection header received
0928 02:50:28.925061 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:28.925064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:28.925111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:28.925158 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:28.925972 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:28.926528 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:28.927357 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:28.927606 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:28.927715 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:28.929205 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:28.929209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:28.929212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:28.929244 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:28.930662 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:28.931130 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:28.931135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:28.931142 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:28.931213 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:28.931549 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:28.931552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:28.931554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:28.931721 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:28.931874 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:28.932137 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:50:28.932229 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:50:28.943992 18880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:28.945276 18880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:28.951459 18880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec27f90afab461280b7d5d02e4ebef5: Generated new TSK 0
I20260928 02:50:28.952267 18880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:28.960486 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18797
W20260928 02:50:28.965795 18782 connection.cc:535] client connection to 127.28.248.190:43031 recv error: Network error: failed to read from TLS socket (remote: 127.28.248.190:43031): Connection reset by peer (error 104)
2026-09-28T02:50:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:29 dist-test-slave-r238 krb5kdc[18896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:29 dist-test-slave-r238 krb5kdc[18896](info): set up 2 sockets
Sep 28 02:50:29 dist-test-slave-r238 krb5kdc[18896](info): commencing operation
krb5kdc: starting...
W20260928 02:50:30.001087 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:30 dist-test-slave-r238 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:30.129734 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:33853
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:30.240533 18909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:30.240895 18909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:30.240980 18909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:30.245044 18909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:30.245153 18909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:30.245193 18909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:30.245231 18909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:30.245267 18909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:30.250510 18909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:33853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:30.251623 18909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:30.252728 18909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:30.261510 18917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:30.261503 18915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:30.261554 18909 server_base.cc:1034] running on GCE node
W20260928 02:50:30.261503 18914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:30.262379 18909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:30.263418 18909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:30.264650 18909 hybrid_clock.cc:648] HybridClock initialized: now 1790563830264634 us; error 26 us; skew 500 ppm
Sep 28 02:50:30 dist-test-slave-r238 krb5kdc[18896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563830, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:30.273418 18909 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:30.274446 18909 webserver.cc:460] Webserver started at http://127.28.248.190:40063/ using document root <none> and password file <none>
I20260928 02:50:30.274991 18909 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:30.275053 18909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:30.275219 18909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:30.277370 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "b5417a890a4f450bbaf24cea25a77904"
format_stamp: "Formatted at 2026-09-28 02:50:30 on dist-test-slave-r238"
server_key: "7f1e93d6472496b2a094c933d3f408dd"
server_key_iv: "20dc5775c76a23d78a986d19eb29d3e5"
server_key_version: "encryptionkey@0"
I20260928 02:50:30.277902 18909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "b5417a890a4f450bbaf24cea25a77904"
format_stamp: "Formatted at 2026-09-28 02:50:30 on dist-test-slave-r238"
server_key: "7f1e93d6472496b2a094c933d3f408dd"
server_key_iv: "20dc5775c76a23d78a986d19eb29d3e5"
server_key_version: "encryptionkey@0"
I20260928 02:50:30.281695 18909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:50:30.284060 18924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:30.285411 18909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:50:30.285517 18909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "b5417a890a4f450bbaf24cea25a77904"
format_stamp: "Formatted at 2026-09-28 02:50:30 on dist-test-slave-r238"
server_key: "7f1e93d6472496b2a094c933d3f408dd"
server_key_iv: "20dc5775c76a23d78a986d19eb29d3e5"
server_key_version: "encryptionkey@0"
I20260928 02:50:30.285599 18909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:30.302987 18909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:30.306185 18909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:30.306346 18909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:30.313660 18909 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:33853
I20260928 02:50:30.313684 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:33853 every 8 connection(s)
I20260928 02:50:30.314599 18909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:30.316355 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 18909
I20260928 02:50:30.316512 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:30.316814 29666 external_mini_cluster.cc:1428] Setting key 5534b9fc6d0ebc988abee319f9de22f7
I20260928 02:50:30.317580 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:30.324324 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904: Bootstrap starting.
I20260928 02:50:30.326799 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:30.318259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50203 (local address 127.28.248.190:33853)
0928 02:50:30.318682 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:30.318700 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:30.318740 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:50:30.319606 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:30.319629 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:30.320065 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:30.320431 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:30.320871 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:30.322339 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:30.323189 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:30.323867 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:30.324017 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:30.324937 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:30.324943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:30.324957 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:30.325425 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:30.325501 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:30.325534 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:30.325672 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:30.325799 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:50:30.325924 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260928 02:50:30.326973 18977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:30.327693 18977 log.cc:824] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:30.329653 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904: No bootstrap required, opened a new log
I20260928 02:50:30.329692 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:30.333741 18977 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5417a890a4f450bbaf24cea25a77904" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33853 } }
I20260928 02:50:30.333961 18977 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:30.334029 18977 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5417a890a4f450bbaf24cea25a77904, State: Initialized, Role: FOLLOWER
I20260928 02:50:30.334466 18977 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5417a890a4f450bbaf24cea25a77904" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33853 } }
I20260928 02:50:30.334620 18977 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:30.334704 18977 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:30.334823 18977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:30.335721 18977 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5417a890a4f450bbaf24cea25a77904" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33853 } }
I20260928 02:50:30.336129 18977 leader_election.cc:302] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5417a890a4f450bbaf24cea25a77904; no voters:
I20260928 02:50:30.336288 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:30.331886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35925 (local address 127.28.248.190:33853)
0928 02:50:30.332038 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:30.332045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:30.332242 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:50:30.332492 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:30.332495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:30.332536 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:30.332645 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:30.332924 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:30.333449 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:30.334452 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:30.334749 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:30.334864 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:30.335860 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:30.335864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:30.335867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:30.335878 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:30.335904 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:30.335916 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:30.336006 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:30.336120 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:50:30.336160 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:50:30.336493 18977 leader_election.cc:288] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:30.336640 18987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:30.336843 18987 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [term 1 LEADER]: Becoming Leader. State: Replica: b5417a890a4f450bbaf24cea25a77904, State: Running, Role: LEADER
I20260928 02:50:30.337221 18987 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5417a890a4f450bbaf24cea25a77904" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33853 } }
I20260928 02:50:30.337785 18977 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:30.338413 18987 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5417a890a4f450bbaf24cea25a77904. Latest consensus state: current_term: 1 leader_uuid: "b5417a890a4f450bbaf24cea25a77904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5417a890a4f450bbaf24cea25a77904" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33853 } } }
I20260928 02:50:30.338634 18987 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:30.338869 18989 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5417a890a4f450bbaf24cea25a77904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5417a890a4f450bbaf24cea25a77904" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33853 } } }
I20260928 02:50:30.339041 18989 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:30.339066 18996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:30.342372 18996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:30.342689 19004 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:50:30.348212 18996 catalog_manager.cc:1347] Generated new cluster ID: 9cfc4f765e5443b4a5cb83d7249859ed
I20260928 02:50:30.348284 18996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:30.358127 18996 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:30.359357 18996 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:30.371791 18996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5417a890a4f450bbaf24cea25a77904: Generated new TSK 0
I20260928 02:50:30.372694 18996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:30.385159 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 18909
W20260928 02:50:30.390555 18894 connection.cc:535] client connection to 127.28.248.190:33853 recv error: Network error: failed to read from TLS socket (remote: 127.28.248.190:33853): Connection reset by peer (error 104)
2026-09-28T02:50:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:30 dist-test-slave-r238 krb5kdc[19008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:30 dist-test-slave-r238 krb5kdc[19008](info): set up 2 sockets
Sep 28 02:50:30 dist-test-slave-r238 krb5kdc[19008](info): commencing operation
krb5kdc: starting...
W20260928 02:50:31.424973 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:31 dist-test-slave-r238 krb5kdc[19008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:31.551695 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:33405
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:43977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:31.664529 19021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:31.664912 19021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:31.665014 19021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:31.669525 19021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:31.669634 19021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:31.669682 19021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:31.669723 19021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:31.669759 19021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:31.675669 19021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:43977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:33405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:31.676939 19021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:31.678048 19021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:50:31.687357 19021 server_base.cc:1034] running on GCE node
W20260928 02:50:31.687232 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:31.687230 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:31.687291 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:31.689258 19021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:31.690763 19021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:31.691955 19021 hybrid_clock.cc:648] HybridClock initialized: now 1790563831691942 us; error 30 us; skew 500 ppm
Sep 28 02:50:31 dist-test-slave-r238 krb5kdc[19008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563831, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:31.695077 19021 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:31.696034 19021 webserver.cc:460] Webserver started at http://127.28.248.190:38337/ using document root <none> and password file <none>
I20260928 02:50:31.696669 19021 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:31.696772 19021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:31.696995 19021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:31.699066 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "9504303a2eb340e5970ea17f2e3fb6c6"
format_stamp: "Formatted at 2026-09-28 02:50:31 on dist-test-slave-r238"
server_key: "f0aa62eeb0750fcfea7054478ca5aaa5"
server_key_iv: "89dcb0c1a0bba57e58357562d6fa9c48"
server_key_version: "encryptionkey@0"
I20260928 02:50:31.699640 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "9504303a2eb340e5970ea17f2e3fb6c6"
format_stamp: "Formatted at 2026-09-28 02:50:31 on dist-test-slave-r238"
server_key: "f0aa62eeb0750fcfea7054478ca5aaa5"
server_key_iv: "89dcb0c1a0bba57e58357562d6fa9c48"
server_key_version: "encryptionkey@0"
I20260928 02:50:31.703444 19021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:50:31.706139 19036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:31.707439 19021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:31.707602 19021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "9504303a2eb340e5970ea17f2e3fb6c6"
format_stamp: "Formatted at 2026-09-28 02:50:31 on dist-test-slave-r238"
server_key: "f0aa62eeb0750fcfea7054478ca5aaa5"
server_key_iv: "89dcb0c1a0bba57e58357562d6fa9c48"
server_key_version: "encryptionkey@0"
I20260928 02:50:31.707717 19021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:31.712227 19021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:31.715118 19021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:31.715315 19021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:31.722321 19021 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:33405
I20260928 02:50:31.722337 19088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:33405 every 8 connection(s)
I20260928 02:50:31.723219 19021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:31.726074 19089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:31.728140 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19021
I20260928 02:50:31.728305 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:31.728629 29666 external_mini_cluster.cc:1428] Setting key da8048c49a5f25e5c05a7e6da68f808f
I20260928 02:50:31.733073 19089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6: Bootstrap starting.
I20260928 02:50:31.733989 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:31.730141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37801 (local address 127.28.248.190:33405)
0928 02:50:31.730598 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:31.730613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:31.730648 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:31.731335 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:31.731352 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:31.731644 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:31.731996 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:31.732821 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:31.732826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:31.732837 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:31.732846 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:31.732900 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:31.732918 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:31.733067 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:31.733136 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:50:31.733293 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":140,"threads_started":1}
I20260928 02:50:31.735468 19089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:31.736348 19089 log.cc:824] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:31.736950 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:31.738621 19089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6: No bootstrap required, opened a new log
I20260928 02:50:31.741187 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:31.739301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45829 (local address 127.28.248.190:33405)
0928 02:50:31.739436 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:31.739440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:31.739668 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:50:31.739799 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:31.739801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:31.739835 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:31.739892 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:31.740587 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:31.740590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:31.740592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:31.740601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:31.740685 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:31.740696 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:31.740771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:31.741002 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 02:50:31.741050 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:31.743029 19089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33405 } }
I20260928 02:50:31.743242 19089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:31.743330 19089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9504303a2eb340e5970ea17f2e3fb6c6, State: Initialized, Role: FOLLOWER
I20260928 02:50:31.743758 19089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33405 } }
I20260928 02:50:31.743880 19089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:31.743928 19089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:31.744000 19089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:31.744979 19089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33405 } }
I20260928 02:50:31.745314 19089 leader_election.cc:302] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9504303a2eb340e5970ea17f2e3fb6c6; no voters:
I20260928 02:50:31.745599 19089 leader_election.cc:288] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:31.745721 19099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:31.746097 19099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9504303a2eb340e5970ea17f2e3fb6c6, State: Running, Role: LEADER
I20260928 02:50:31.746434 19099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33405 } }
I20260928 02:50:31.746807 19089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:31.748064 19101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9504303a2eb340e5970ea17f2e3fb6c6. Latest consensus state: current_term: 1 leader_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33405 } } }
I20260928 02:50:31.748111 19100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9504303a2eb340e5970ea17f2e3fb6c6" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33405 } } }
I20260928 02:50:31.748202 19101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:31.748206 19100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:31.748726 19110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:31.751509 19110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:31.756532 19110 catalog_manager.cc:1347] Generated new cluster ID: 692a0eec0a8e434f9cd1422a6b08cf0d
I20260928 02:50:31.756597 19110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:31.769614 19110 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:31.769941 19110 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:31.777419 19110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9504303a2eb340e5970ea17f2e3fb6c6: Generated new TSK 0
I20260928 02:50:31.778106 19110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:31.785873 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19021
2026-09-28T02:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:31 dist-test-slave-r238 krb5kdc[19120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:31 dist-test-slave-r238 krb5kdc[19120](info): set up 2 sockets
Sep 28 02:50:31 dist-test-slave-r238 krb5kdc[19120](info): commencing operation
krb5kdc: starting...
W20260928 02:50:32.845010 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:32 dist-test-slave-r238 krb5kdc[19120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:32.967790 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38931
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:38931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:33.094612 19133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:33.094939 19133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:33.095057 19133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:33.099179 19133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:33.099274 19133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:33.099313 19133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:33.099367 19133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:33.099404 19133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:33.104732 19133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:38931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:38931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:33.105818 19133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:33.106830 19133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:33.115387 19138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:33.115402 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:33.115392 19141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:33.115661 19133 server_base.cc:1034] running on GCE node
I20260928 02:50:33.116293 19133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:33.117409 19133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:33.118605 19133 hybrid_clock.cc:648] HybridClock initialized: now 1790563833118586 us; error 31 us; skew 500 ppm
Sep 28 02:50:33 dist-test-slave-r238 krb5kdc[19120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563833, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:33.121263 19133 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:33.122082 19133 webserver.cc:460] Webserver started at http://127.28.248.190:39099/ using document root <none> and password file <none>
I20260928 02:50:33.122622 19133 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:33.122714 19133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:33.122987 19133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:33.125016 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "ac620322769a4eac92b92d2d55e1de90"
format_stamp: "Formatted at 2026-09-28 02:50:33 on dist-test-slave-r238"
server_key: "9af14df4351810c8151d7fbe7e263b37"
server_key_iv: "1324e5f9b46f462e7a3a00353d3d74cf"
server_key_version: "encryptionkey@0"
I20260928 02:50:33.125576 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "ac620322769a4eac92b92d2d55e1de90"
format_stamp: "Formatted at 2026-09-28 02:50:33 on dist-test-slave-r238"
server_key: "9af14df4351810c8151d7fbe7e263b37"
server_key_iv: "1324e5f9b46f462e7a3a00353d3d74cf"
server_key_version: "encryptionkey@0"
I20260928 02:50:33.129091 19133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:50:33.131433 19148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:33.132676 19133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:33.132812 19133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "ac620322769a4eac92b92d2d55e1de90"
format_stamp: "Formatted at 2026-09-28 02:50:33 on dist-test-slave-r238"
server_key: "9af14df4351810c8151d7fbe7e263b37"
server_key_iv: "1324e5f9b46f462e7a3a00353d3d74cf"
server_key_version: "encryptionkey@0"
I20260928 02:50:33.132915 19133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:33.150774 19133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:33.153559 19133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:33.153755 19133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:33.160358 19133 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:38931
I20260928 02:50:33.160416 19200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:38931 every 8 connection(s)
I20260928 02:50:33.161394 19133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:33.163676 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19133
I20260928 02:50:33.163832 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:33.164136 29666 external_mini_cluster.cc:1428] Setting key b0db67de1f323ae23f375594540c111d
I20260928 02:50:33.164410 19201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:33.170877 19201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90: Bootstrap starting.
Sep 28 02:50:33 dist-test-slave-r238 krb5kdc[19120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:33.173477 19201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:33.174336 19201 log.cc:824] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:33.176298 19201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90: No bootstrap required, opened a new log
I20260928 02:50:33.178265 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:33.165639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57527 (local address 127.28.248.190:38931)
0928 02:50:33.166074 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:33.166088 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:33.166123 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:33.166765 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:33.166781 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:33.167125 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:33.167463 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:33.168279 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:33.169367 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:33.170235 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:33.170706 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:33.170888 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:33.173640 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:33.173646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:33.173668 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:33.173696 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:33.175436 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:33.175883 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:33.175888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:33.175893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:33.176025 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:33.176324 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:33.176332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:33.176333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:33.176825 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:33.177034 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:33.177279 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:50:33.177572 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260928 02:50:33.179971 19201 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac620322769a4eac92b92d2d55e1de90" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38931 } }
I20260928 02:50:33.180251 19201 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:33.180326 19201 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac620322769a4eac92b92d2d55e1de90, State: Initialized, Role: FOLLOWER
I20260928 02:50:33.180804 19201 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac620322769a4eac92b92d2d55e1de90" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38931 } }
I20260928 02:50:33.180958 19201 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:33.181028 19201 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:33.181049 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:33.181130 19201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:33.182009 19201 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac620322769a4eac92b92d2d55e1de90" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38931 } }
I20260928 02:50:33.182421 19201 leader_election.cc:302] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac620322769a4eac92b92d2d55e1de90; no voters:
I20260928 02:50:33.182755 19201 leader_election.cc:288] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:33.182868 19210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:33.183108 19210 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [term 1 LEADER]: Becoming Leader. State: Replica: ac620322769a4eac92b92d2d55e1de90, State: Running, Role: LEADER
I20260928 02:50:33.183425 19210 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac620322769a4eac92b92d2d55e1de90" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38931 } }
I20260928 02:50:33.184072 19201 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:33.184512 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac620322769a4eac92b92d2d55e1de90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac620322769a4eac92b92d2d55e1de90" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38931 } } }
I20260928 02:50:33.184644 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:33.184841 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac620322769a4eac92b92d2d55e1de90. Latest consensus state: current_term: 1 leader_uuid: "ac620322769a4eac92b92d2d55e1de90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac620322769a4eac92b92d2d55e1de90" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 38931 } } }
I20260928 02:50:33.184940 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:33.185187 19215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:33.188642 19215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:33.192134 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:33.183258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35817 (local address 127.28.248.190:38931)
0928 02:50:33.183355 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:33.183359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:33.183373 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:33.183437 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:33.183439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:33.183479 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:33.183562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:33.184421 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:33.185274 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:33.186521 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:33.186746 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:33.186857 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:33.188563 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:33.188567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:33.188569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:33.188595 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:33.190377 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:33.190857 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:33.190860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:33.190862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:33.190959 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:33.191270 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:33.191273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:33.191274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:33.191476 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:33.191612 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:33.191879 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:50:33.191999 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:50:33.194621 19215 catalog_manager.cc:1347] Generated new cluster ID: 7397e6a0a6f54a48ab897db0e30701fb
I20260928 02:50:33.194691 19215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:33.207731 19215 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:33.208753 19215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:33.214419 19215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac620322769a4eac92b92d2d55e1de90: Generated new TSK 0
I20260928 02:50:33.215111 19215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:33.222129 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19133
2026-09-28T02:50:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:33 dist-test-slave-r238 krb5kdc[19232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:33 dist-test-slave-r238 krb5kdc[19232](info): set up 2 sockets
Sep 28 02:50:33 dist-test-slave-r238 krb5kdc[19232](info): commencing operation
krb5kdc: starting...
W20260928 02:50:34.268981 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:34 dist-test-slave-r238 krb5kdc[19232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:34.392554 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:40415
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:40415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:34.501292 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:34.501536 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:34.501608 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:34.505611 19245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:34.505667 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:34.505687 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:34.505703 19245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:34.505764 19245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:34.510892 19245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:40415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:34.511948 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:34.512955 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:34.521814 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:34.521924 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:34.521886 19245 server_base.cc:1034] running on GCE node
W20260928 02:50:34.521813 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:34.522668 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:34.523921 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:34.525090 19245 hybrid_clock.cc:648] HybridClock initialized: now 1790563834525057 us; error 45 us; skew 500 ppm
Sep 28 02:50:34 dist-test-slave-r238 krb5kdc[19232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563834, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:34.527968 19245 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:34.529196 19245 webserver.cc:460] Webserver started at http://127.28.248.190:43855/ using document root <none> and password file <none>
I20260928 02:50:34.529894 19245 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:34.530006 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:34.530265 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:34.532480 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "8f50060d4a29455eac104adf72f8dca4"
format_stamp: "Formatted at 2026-09-28 02:50:34 on dist-test-slave-r238"
server_key: "86382044f3fe09f2cc72fe60d344d8bb"
server_key_iv: "ec1b4149cb5a4f951a0d2092f7435516"
server_key_version: "encryptionkey@0"
I20260928 02:50:34.533169 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "8f50060d4a29455eac104adf72f8dca4"
format_stamp: "Formatted at 2026-09-28 02:50:34 on dist-test-slave-r238"
server_key: "86382044f3fe09f2cc72fe60d344d8bb"
server_key_iv: "ec1b4149cb5a4f951a0d2092f7435516"
server_key_version: "encryptionkey@0"
I20260928 02:50:34.537190 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:50:34.539695 19260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:34.541067 19245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:34.541220 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "8f50060d4a29455eac104adf72f8dca4"
format_stamp: "Formatted at 2026-09-28 02:50:34 on dist-test-slave-r238"
server_key: "86382044f3fe09f2cc72fe60d344d8bb"
server_key_iv: "ec1b4149cb5a4f951a0d2092f7435516"
server_key_version: "encryptionkey@0"
I20260928 02:50:34.541334 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:34.554904 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:34.557751 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:34.557991 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:34.564980 19245 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:40415
I20260928 02:50:34.565002 19312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:40415 every 8 connection(s)
I20260928 02:50:34.565943 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:34.568533 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19245
I20260928 02:50:34.568702 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:34.568809 19313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:34.569020 29666 external_mini_cluster.cc:1428] Setting key ac120a6ed9d423d8e658d44af96ef291
I20260928 02:50:34.575177 19313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4: Bootstrap starting.
I20260928 02:50:34.577580 19313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:34.578111 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:34.570465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43609 (local address 127.28.248.190:40415)
0928 02:50:34.570945 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:34.570960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:34.570995 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:50:34.571626 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:34.571642 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:34.572038 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:34.572398 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:34.572849 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:34.573875 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:34.574781 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:34.575296 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:34.575485 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:34.576351 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:34.576358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:34.576371 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:34.576813 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:34.576872 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:34.576896 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:34.576997 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:34.577153 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:50:34.577349 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":146,"threads_started":1}
I20260928 02:50:34.578361 19313 log.cc:824] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:34.580257 19313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4: No bootstrap required, opened a new log
I20260928 02:50:34.580818 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:34.584151 19313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f50060d4a29455eac104adf72f8dca4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40415 } }
I20260928 02:50:34.584465 19313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:34.584539 19313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f50060d4a29455eac104adf72f8dca4, State: Initialized, Role: FOLLOWER
I20260928 02:50:34.585031 19313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f50060d4a29455eac104adf72f8dca4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40415 } }
I20260928 02:50:34.585179 19313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:34.585244 19313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:34.585366 19313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:34.586524 19313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f50060d4a29455eac104adf72f8dca4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40415 } }
I20260928 02:50:34.586913 19313 leader_election.cc:302] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f50060d4a29455eac104adf72f8dca4; no voters:
I20260928 02:50:34.587069 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:34.582844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50385 (local address 127.28.248.190:40415)
0928 02:50:34.582958 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:34.582961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:34.583097 (+ 136us) server_negotiation.cc:407] Connection header received
0928 02:50:34.583213 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:34.583215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:34.583249 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:34.583311 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:34.583610 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:34.584152 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:34.585327 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:34.585646 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:34.585726 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:34.586501 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:34.586504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:34.586506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:34.586516 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:34.586537 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:34.586547 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:34.586604 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:34.586893 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:50:34.586949 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:50:34.587252 19313 leader_election.cc:288] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:34.587333 19323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:34.587565 19323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f50060d4a29455eac104adf72f8dca4, State: Running, Role: LEADER
I20260928 02:50:34.587857 19323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f50060d4a29455eac104adf72f8dca4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40415 } }
I20260928 02:50:34.588426 19313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:34.589011 19325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f50060d4a29455eac104adf72f8dca4. Latest consensus state: current_term: 1 leader_uuid: "8f50060d4a29455eac104adf72f8dca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f50060d4a29455eac104adf72f8dca4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40415 } } }
I20260928 02:50:34.589134 19325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:34.589497 19324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f50060d4a29455eac104adf72f8dca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f50060d4a29455eac104adf72f8dca4" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 40415 } } }
I20260928 02:50:34.589610 19324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4 [sys.catalog]: This master's current role is: LEADER
W20260928 02:50:34.593195 19338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:34.593295 19338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:34.593376 19330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:34.594074 19330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:34.599205 19330 catalog_manager.cc:1347] Generated new cluster ID: cc4a65ea7cb54110985dc1ed34ce49f2
I20260928 02:50:34.599265 19330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:34.618283 19330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:34.619554 19330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:34.625192 19330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f50060d4a29455eac104adf72f8dca4: Generated new TSK 0
I20260928 02:50:34.625942 19330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:34.633445 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19245
2026-09-28T02:50:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:34 dist-test-slave-r238 krb5kdc[19344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:34 dist-test-slave-r238 krb5kdc[19344](info): set up 2 sockets
Sep 28 02:50:34 dist-test-slave-r238 krb5kdc[19344](info): commencing operation
krb5kdc: starting...
W20260928 02:50:35.669046 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:35 dist-test-slave-r238 krb5kdc[19344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:35.792075 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46029
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:35.900550 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:35.900878 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:35.900967 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:35.904979 19357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:35.905067 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:35.905104 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:35.905161 19357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:35.905202 19357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:35.910265 19357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:46029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:35.911309 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:35.912338 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:35.920681 19362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:35.920694 19365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:35.920778 19357 server_base.cc:1034] running on GCE node
W20260928 02:50:35.920694 19363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:35.921496 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:35.922533 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:35.923730 19357 hybrid_clock.cc:648] HybridClock initialized: now 1790563835923689 us; error 61 us; skew 500 ppm
Sep 28 02:50:35 dist-test-slave-r238 krb5kdc[19344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563835, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:35.931866 19357 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:35.932900 19357 webserver.cc:460] Webserver started at http://127.28.248.190:44455/ using document root <none> and password file <none>
I20260928 02:50:35.933491 19357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:35.933609 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:35.933841 19357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:35.935914 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a"
format_stamp: "Formatted at 2026-09-28 02:50:35 on dist-test-slave-r238"
server_key: "264bddf0629eb4a7c0745f049158c1fe"
server_key_iv: "827fa8830c6d51c469114fff777197bd"
server_key_version: "encryptionkey@0"
I20260928 02:50:35.936479 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a"
format_stamp: "Formatted at 2026-09-28 02:50:35 on dist-test-slave-r238"
server_key: "264bddf0629eb4a7c0745f049158c1fe"
server_key_iv: "827fa8830c6d51c469114fff777197bd"
server_key_version: "encryptionkey@0"
I20260928 02:50:35.940291 19357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:50:35.942890 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:35.944175 19357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:35.944345 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a"
format_stamp: "Formatted at 2026-09-28 02:50:35 on dist-test-slave-r238"
server_key: "264bddf0629eb4a7c0745f049158c1fe"
server_key_iv: "827fa8830c6d51c469114fff777197bd"
server_key_version: "encryptionkey@0"
I20260928 02:50:35.944463 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:35.949172 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:35.952042 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:35.952229 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:35.959049 19357 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:46029
I20260928 02:50:35.959079 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:46029 every 8 connection(s)
I20260928 02:50:35.959936 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:35.962610 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:35.967772 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19357
I20260928 02:50:35.967886 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:35.968171 29666 external_mini_cluster.cc:1428] Setting key 0c61f7da48b49e8dea5e752ebb72ebd4
I20260928 02:50:35.968154 19425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a: Bootstrap starting.
I20260928 02:50:35.970655 19425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:35.971398 19425 log.cc:824] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:35.973459 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:35.969583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39681 (local address 127.28.248.190:46029)
0928 02:50:35.970014 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:35.970028 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:35.970062 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:50:35.970726 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:35.970745 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:35.971129 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:35.971477 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:35.972297 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:35.972301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:35.972314 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:35.972324 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:35.972376 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:35.972394 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:35.972510 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:35.972631 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:50:35.972793 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":136,"threads_started":1}
I20260928 02:50:35.973547 19425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a: No bootstrap required, opened a new log
I20260928 02:50:35.976400 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:35.977768 19425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46029 } }
I20260928 02:50:35.978029 19425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:50:35.978109 19425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e3c5feaa7b4de4a18f4fdb78ce023a, State: Initialized, Role: FOLLOWER
I20260928 02:50:35.978572 19425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46029 } }
I20260928 02:50:35.978781 19425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:50:35.978878 19425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:50:35.978982 19425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:35.979890 19425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46029 } }
I20260928 02:50:35.980268 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:35.978562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58791 (local address 127.28.248.190:46029)
0928 02:50:35.978688 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:35.978694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:35.978872 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:50:35.979019 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:35.979023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:35.979084 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:35.979157 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:35.979890 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:35.979894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:35.979896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:50:35.979908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:35.979934 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:50:35.979947 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:35.980044 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:35.980114 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:50:35.980154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:50:35.980275 19425 leader_election.cc:302] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e3c5feaa7b4de4a18f4fdb78ce023a; no voters:
I20260928 02:50:35.980664 19425 leader_election.cc:288] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:50:35.980801 19435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:35.981045 19435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [term 1 LEADER]: Becoming Leader. State: Replica: 59e3c5feaa7b4de4a18f4fdb78ce023a, State: Running, Role: LEADER
I20260928 02:50:35.981402 19435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46029 } }
I20260928 02:50:35.981909 19425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:35.983140 19437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46029 } } }
I20260928 02:50:35.983162 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e3c5feaa7b4de4a18f4fdb78ce023a. Latest consensus state: current_term: 1 leader_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e3c5feaa7b4de4a18f4fdb78ce023a" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 46029 } } }
I20260928 02:50:35.983307 19437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:35.983307 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:35.983752 19448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:35.987118 19448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:35.992571 19448 catalog_manager.cc:1347] Generated new cluster ID: a1539fcf67ee4cb89b8e92b2c99ec018
I20260928 02:50:35.992700 19448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:36.000223 19448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:36.000598 19448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:36.006042 19448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e3c5feaa7b4de4a18f4fdb78ce023a: Generated new TSK 0
I20260928 02:50:36.006695 19448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:50:36.018725 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19357
2026-09-28T02:50:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1386 ms)
[----------] 18 tests from AuthTokenIssuingTest (25624 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:36 dist-test-slave-r238 krb5kdc[19456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:36 dist-test-slave-r238 krb5kdc[19456](info): set up 2 sockets
Sep 28 02:50:36 dist-test-slave-r238 krb5kdc[19456](info): commencing operation
krb5kdc: starting...
W20260928 02:50:37.085088 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:37.209573 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:34631
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:34631,127.28.248.189:45439,127.28.248.188:40417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:37.316948 19469 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:50:37.317214 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:37.317237 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:37.317274 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:37.321111 19469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:37.321168 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:37.321188 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:37.321205 19469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:37.321220 19469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:37.326321 19469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:34631,127.28.248.189:45439,127.28.248.188:40417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:37.327231 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:37.328125 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:37.336335 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:37.336359 19474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:37.336371 19475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:37.336340 19469 server_base.cc:1034] running on GCE node
I20260928 02:50:37.337172 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:37.338291 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:37.339481 19469 hybrid_clock.cc:648] HybridClock initialized: now 1790563837339453 us; error 39 us; skew 500 ppm
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:37.342269 19469 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:37.343081 19469 webserver.cc:460] Webserver started at http://127.28.248.190:34931/ using document root <none> and password file <none>
I20260928 02:50:37.343586 19469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:37.343643 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:37.343801 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:37.345856 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "6cb2c42a657c49a68c6cc7b675fee0aa"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "75a429d90e23c8bc31e96f8b2d21696b"
server_key_iv: "9170abf6f0596d5a65a1ac9dfd912645"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.346369 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "6cb2c42a657c49a68c6cc7b675fee0aa"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "75a429d90e23c8bc31e96f8b2d21696b"
server_key_iv: "9170abf6f0596d5a65a1ac9dfd912645"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.349881 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:50:37.352344 19484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:37.353652 19469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:50:37.353755 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "6cb2c42a657c49a68c6cc7b675fee0aa"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "75a429d90e23c8bc31e96f8b2d21696b"
server_key_iv: "9170abf6f0596d5a65a1ac9dfd912645"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.353843 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:37.390708 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:37.393860 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:37.394029 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:37.401170 19469 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:34631
I20260928 02:50:37.401480 19536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:34631 every 8 connection(s)
I20260928 02:50:37.402166 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:37.404974 19537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:37.406138 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19469
I20260928 02:50:37.406289 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:37.406598 29666 external_mini_cluster.cc:1428] Setting key 5f8e03f32409e2961bc345a1070b4341
I20260928 02:50:37.409806 19537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.248.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.189@KRBTEST.COM" created.
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:37.425143 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.412278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34631 (local address 127.0.0.1:37198)
0928 02:50:37.412883 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.412919 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.413750 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.414432 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.414437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.414882 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.415119 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.415132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.416265 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.416268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.417054 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.417062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.417837 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.417841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.417964 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.418569 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.418585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.419991 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.422355 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.422363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.422370 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.422593 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.422999 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.423002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.423003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.423104 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.423800 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.423805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.423922 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.424058 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.424297 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"spinlock_wait_cycles":9600,"thread_start_us":144,"threads_started":1}
I20260928 02:50:37.425150 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.412277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.28.248.190:34631)
0928 02:50:37.412716 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.412738 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.413027 (+ 289us) server_negotiation.cc:407] Connection header received
0928 02:50:37.414062 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.414066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.414244 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.414438 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.415291 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.416142 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.417181 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.417654 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.417817 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.420124 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.420129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.420138 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.420158 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.422225 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.422707 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.422716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.422720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.422871 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.423198 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.423203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.423205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.423691 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.423835 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.424182 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.424397 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":275,"thread_start_us":138,"threads_started":1}
I20260928 02:50:37.427644 19537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:37.428514 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.428258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41278)
0928 02:50:37.428377 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.428460 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:50:37.428788 19485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
I20260928 02:50:37.429632 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.429437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41292)
0928 02:50:37.429542 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.429584 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:50:37.429858 19537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:45439: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab.
Entry for principal kudu/127.28.248.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab.
I20260928 02:50:37.436817 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:45439
--webserver_interface=127.28.248.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab
--principal=kudu/127.28.248.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:34631,127.28.248.189:45439,127.28.248.188:40417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:50:37.479180 19537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } attempt: 1
I20260928 02:50:37.480106 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.479803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41302)
0928 02:50:37.479966 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.480029 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:50:37.480906 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.480710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41318)
0928 02:50:37.480806 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.480851 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:50:37.481065 19537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:45439: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
W20260928 02:50:37.547595 19544 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:50:37.547884 19544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:37.547945 19544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:37.548030 19544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:50:37.549230 19537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } attempt: 2
I20260928 02:50:37.550087 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.549796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41332)
0928 02:50:37.549950 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.550014 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:50:37.550781 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.550601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41346)
0928 02:50:37.550691 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.550734 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 02:50:37.550923 19537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:45439: Network error: Client connection negotiation failed: client connection to 127.28.248.189:45439: connect: Connection refused (error 111)
W20260928 02:50:37.552780 19544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:37.552834 19544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:37.552875 19544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:37.552917 19544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:37.552974 19544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:37.558442 19544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:34631,127.28.248.189:45439,127.28.248.188:40417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab
--principal=kudu/127.28.248.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.248.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:37.559482 19544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:37.560511 19544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:37.568967 19550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:37.568997 19552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:37.569010 19549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:37.569059 19544 server_base.cc:1034] running on GCE node
I20260928 02:50:37.569687 19544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:37.570789 19544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:37.571972 19544 hybrid_clock.cc:648] HybridClock initialized: now 1790563837571934 us; error 47 us; skew 500 ppm
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:37.574781 19544 init.cc:377] Logged in from keytab as kudu/127.28.248.189@KRBTEST.COM (short username kudu)
I20260928 02:50:37.575690 19544 webserver.cc:460] Webserver started at http://127.28.248.189:42939/ using document root <none> and password file <none>
I20260928 02:50:37.576239 19544 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:37.576309 19544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:37.576573 19544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:37.578598 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data/instance:
uuid: "7ffc74c8fe504188bd10a99097fdfe69"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "a4d9e8e276bcd5f0858b506e3af7027d"
server_key_iv: "dbaf174cc54207c922409f69eb26e539"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.579173 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal/instance:
uuid: "7ffc74c8fe504188bd10a99097fdfe69"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "a4d9e8e276bcd5f0858b506e3af7027d"
server_key_iv: "dbaf174cc54207c922409f69eb26e539"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.582912 19544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 02:50:37.585258 19559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:37.586560 19544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 02:50:37.586700 19544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal
uuid: "7ffc74c8fe504188bd10a99097fdfe69"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "a4d9e8e276bcd5f0858b506e3af7027d"
server_key_iv: "dbaf174cc54207c922409f69eb26e539"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.586805 19544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:37.606277 19544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:37.609769 19544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:37.610065 19544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:37.617341 19544 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.189:45439
I20260928 02:50:37.617648 19611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.189:45439 every 8 connection(s)
I20260928 02:50:37.618364 19544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
I20260928 02:50:37.621158 19612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:37.622936 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19544
I20260928 02:50:37.623095 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-1/wal/instance
I20260928 02:50:37.623366 29666 external_mini_cluster.cc:1428] Setting key 8ef3c2c85c96ffdaafa17a4410dd2857
I20260928 02:50:37.625089 19612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.248.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.188@KRBTEST.COM" created.
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:37.639294 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.627269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.28.248.190:34631)
0928 02:50:37.627430 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.627435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.628341 (+ 906us) server_negotiation.cc:407] Connection header received
0928 02:50:37.629337 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.629344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.629411 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.629538 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.630705 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.631621 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.632814 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.633050 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.633135 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.635636 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.635644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.635646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.635669 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.637200 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.637727 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.637732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.637734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.637814 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.638163 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.638167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.638169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.638352 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.638465 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.638974 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.639097 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:50:37.640087 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.627688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34631 (local address 127.0.0.1:37200)
0928 02:50:37.628157 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.628211 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.629124 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.629683 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.629690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.630075 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.630545 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.630559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.631778 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.631781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.632708 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.632718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.633181 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.633184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.633345 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.634010 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.634027 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.635467 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.637337 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.637343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.637358 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.637620 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.637908 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.637912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.637914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.638063 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.638466 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.638474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.638727 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.638920 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.639256 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
I20260928 02:50:37.641755 19612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:37.642113 19537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } attempt: 3
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab.
Entry for principal kudu/127.28.248.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab.
I20260928 02:50:37.652462 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:40417
--webserver_interface=127.28.248.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab
--principal=kudu/127.28.248.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:34631,127.28.248.189:45439,127.28.248.188:40417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:50:37.652742 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.642527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41364)
0928 02:50:37.642694 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.642708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.642786 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.643222 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.643225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.643399 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.643931 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.643939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.644773 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.644778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.645464 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.645474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.645979 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.645983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.646085 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.646736 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.646747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.648462 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.650822 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.650828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.650832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.651063 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.651378 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.651380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.651382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.651502 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.652210 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.652214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.652324 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.652390 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.652552 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 02:50:37.652838 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.642276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41356)
0928 02:50:37.642375 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.642390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.642488 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.643188 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.643191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.643362 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.643448 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.643454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.644522 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.644525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.645265 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.645274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.645943 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.645954 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.646043 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.646584 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.646604 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.647932 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.650868 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.650871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.650873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.651058 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.651471 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.651474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.651476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.651578 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.652207 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.652213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.652330 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.652458 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.652600 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:50:37.653126 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.642455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.28.248.189:45439)
0928 02:50:37.642791 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.642804 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.642827 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:50:37.642914 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.642919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.643074 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.643212 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.643589 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.644367 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.645424 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.645831 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.645946 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.648069 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.648077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.648090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.648110 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.650729 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.651170 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.651178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.651179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.651370 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.651676 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.651681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.651682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.652083 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.652224 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.652568 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.652735 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":240,"spinlock_wait_cycles":4352,"thread_start_us":134,"threads_started":1}
I20260928 02:50:37.653306 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.642592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.28.248.189:45439)
0928 02:50:37.642849 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.642858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.642871 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:37.642947 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.642950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.643106 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.643212 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.644054 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.644664 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.645578 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.645855 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.645994 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.648663 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.648669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.648672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.648700 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.650709 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.651176 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.651179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.651181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.651272 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.651626 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.651629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.651634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.652083 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.652930 (+ 847us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.653009 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.653120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":198,"spinlock_wait_cycles":36736,"thread_start_us":100,"threads_started":1}
I20260928 02:50:37.655083 19612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:37.655112 19537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:37.655788 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.655524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47048)
0928 02:50:37.655649 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.655701 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:50:37.655879 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.655553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47064)
0928 02:50:37.655725 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.655802 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 02:50:37.656072 19563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
I20260928 02:50:37.656656 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.656267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47068)
0928 02:50:37.656454 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.656514 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 02:50:37.656787 19537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:40417: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
I20260928 02:50:37.657197 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.656970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47080)
0928 02:50:37.657074 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.657121 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:50:37.657330 19612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:40417: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
I20260928 02:50:37.687976 19537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } attempt: 1
I20260928 02:50:37.688936 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.688597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47096)
0928 02:50:37.688792 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.688857 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:50:37.689687 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.689475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47112)
0928 02:50:37.689585 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.689636 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:50:37.689867 19537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:40417: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
I20260928 02:50:37.706597 19612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } attempt: 1
I20260928 02:50:37.707428 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.707158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47120)
0928 02:50:37.707290 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.707358 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:50:37.708225 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.707967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47134)
0928 02:50:37.708102 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.708156 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:50:37.708400 19612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:40417: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
I20260928 02:50:37.765057 19537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } attempt: 2
I20260928 02:50:37.765940 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.765635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47150)
0928 02:50:37.765791 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.765859 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:50:37.766196 19619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:50:37.766577 19619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:37.766630 19619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:37.766705 19619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:50:37.766747 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47162)
0928 02:50:37.766613 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.766670 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:50:37.766938 19537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:40417: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
W20260928 02:50:37.770991 19619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:37.771046 19619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:37.771066 19619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:37.771082 19619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:37.771142 19619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:37.776458 19619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:34631,127.28.248.189:45439,127.28.248.188:40417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab
--principal=kudu/127.28.248.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:40417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.248.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:37.776647 19612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } attempt: 2
I20260928 02:50:37.777419 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.777129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47170)
0928 02:50:37.777280 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.777350 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:50:37.777762 19619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:37.778146 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.777957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47184)
0928 02:50:37.778054 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.778098 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:50:37.778374 19612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:40417: Network error: Client connection negotiation failed: client connection to 127.28.248.188:40417: connect: Connection refused (error 111)
I20260928 02:50:37.778937 19619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:37.787493 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:37.787576 19619 server_base.cc:1034] running on GCE node
W20260928 02:50:37.787524 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:37.787489 19628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:37.788288 19619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:37.789469 19619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:37.790660 19619 hybrid_clock.cc:648] HybridClock initialized: now 1790563837790626 us; error 49 us; skew 500 ppm
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:37.793535 19619 init.cc:377] Logged in from keytab as kudu/127.28.248.188@KRBTEST.COM (short username kudu)
I20260928 02:50:37.794467 19619 webserver.cc:460] Webserver started at http://127.28.248.188:41715/ using document root <none> and password file <none>
I20260928 02:50:37.794986 19619 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:37.795042 19619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:37.795205 19619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:37.797241 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data/instance:
uuid: "a70241bd90e04affbc2b56de39331b99"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "ae8a0bc287984faa5c4d23eda4eaecb0"
server_key_iv: "46e8c8dba3bebf4ca1011b206e560d56"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.797748 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal/instance:
uuid: "a70241bd90e04affbc2b56de39331b99"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "ae8a0bc287984faa5c4d23eda4eaecb0"
server_key_iv: "46e8c8dba3bebf4ca1011b206e560d56"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.801383 19619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:50:37.803793 19635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:37.805089 19619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:50:37.805223 19619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal
uuid: "a70241bd90e04affbc2b56de39331b99"
format_stamp: "Formatted at 2026-09-28 02:50:37 on dist-test-slave-r238"
server_key: "ae8a0bc287984faa5c4d23eda4eaecb0"
server_key_iv: "46e8c8dba3bebf4ca1011b206e560d56"
server_key_version: "encryptionkey@0"
I20260928 02:50:37.805332 19619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:37.818611 19619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:37.821874 19619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:37.822082 19619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:37.829077 19619 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.188:40417
I20260928 02:50:37.829377 19687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.188:40417 every 8 connection(s)
I20260928 02:50:37.830047 19619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
I20260928 02:50:37.830466 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19619
I20260928 02:50:37.830623 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563560542449-29666-0/minicluster-data/master-2/wal/instance
I20260928 02:50:37.830922 29666 external_mini_cluster.cc:1428] Setting key 84a021e8adb26580766709c78ec0c69a
I20260928 02:50:37.833444 19688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:37.837318 19688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:37.847770 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.832547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.28.248.188:40417)
0928 02:50:37.833830 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.833856 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.833899 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:50:37.834812 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.834831 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.835150 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.835505 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.836356 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.837428 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.838380 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.838929 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.839120 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.842412 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.842420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.842431 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.842465 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.844425 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.844996 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.845002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.845010 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.845153 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.845445 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.845451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.845453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.846010 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.846227 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.846515 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.846804 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":168,"threads_started":1}
I20260928 02:50:37.848469 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.838928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:34631 (local address 127.0.0.1:37202)
0928 02:50:37.839388 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.839436 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.839567 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.839985 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.839989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.840306 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.840424 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.840429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.841516 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.841520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.842319 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.842329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.842924 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.842931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.843083 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.843572 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.843593 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.845044 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.846915 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.846923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.846927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.847139 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.847427 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.847429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.847430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.847503 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.847938 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.847946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.848068 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.848204 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.848332 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20260928 02:50:37.848544 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.839030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.28.248.190:34631)
0928 02:50:37.839169 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.839174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.839541 (+ 367us) server_negotiation.cc:407] Connection header received
0928 02:50:37.839680 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.839685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.839750 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.839942 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.840581 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.841384 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.842417 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.842805 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.842896 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.845187 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.845194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.845198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.845229 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.846807 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.847236 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.847239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.847241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.847338 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.847588 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.847591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.847593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.847817 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.847919 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.848287 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.848405 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:50:37.849687 19688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:37.858662 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.850156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:45439 (local address 127.0.0.1:41372)
0928 02:50:37.850578 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.850588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.850644 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.850850 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.850852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.850999 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.851084 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.851089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.852025 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.852028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.852530 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.852538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.853115 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.853119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.853221 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.853763 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.853783 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.854967 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.856978 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.856984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.856986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.857227 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.857534 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.857539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.857543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.857637 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.858077 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.858081 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.858192 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.858350 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.858483 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260928 02:50:37.859028 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.850275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.28.248.189:45439)
0928 02:50:37.850443 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.850448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.850645 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:50:37.850700 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.850703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.850754 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.850845 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.851199 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.851886 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.852694 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.852983 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.853101 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.855106 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.855112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.855117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.855149 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.856865 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.857340 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.857343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.857345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.857434 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.857734 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.857740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.857744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.857966 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.858062 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.858431 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.858571 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:50:37.859195 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.850196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.28.248.190:34631)
0928 02:50:37.850295 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.850299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.850440 (+ 141us) server_negotiation.cc:407] Connection header received
0928 02:50:37.850544 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.850546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.850580 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.850659 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.851432 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.851900 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.852761 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.852965 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.853069 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.855653 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.855657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.855659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.855683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.857174 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.857611 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.857613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.857614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.857703 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.858063 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.858066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.858068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.858294 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.858527 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.858897 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.859020 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:50:37.859992 19688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
I20260928 02:50:37.866106 19537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } attempt: 3
I20260928 02:50:37.869226 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.860534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47200)
0928 02:50:37.860727 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.860741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.860822 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.861075 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.861078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.861244 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.861308 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.861320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.862138 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.862143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.862669 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.862677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.863160 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.863164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.863291 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.863847 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.863862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.865221 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.867561 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.867566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.867568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.867817 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.868138 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.868141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.868143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.868231 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.868747 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.868750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.868837 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.868941 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.869071 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":25}
I20260928 02:50:37.869509 19612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } attempt: 3
I20260928 02:50:37.869761 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.860747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.28.248.188:40417)
0928 02:50:37.860881 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.860886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.860900 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:37.860930 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.860933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.860974 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.861064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.861434 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.862017 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.862788 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.863049 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.863164 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.865376 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.865379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.865381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.865401 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.867038 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.867937 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.867943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.867946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.868031 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.868335 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.868339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.868341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.868578 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.868760 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.869504 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.869612 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":41}
I20260928 02:50:37.871766 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.860525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.28.248.189:45439)
0928 02:50:37.860663 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.860668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.860681 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:37.860766 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.860770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.860808 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.860898 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.861535 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.862021 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.862760 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.862989 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.863094 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.865996 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.866000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.866003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.866026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.868457 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.868909 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.868915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.868919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.869003 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.869310 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.869315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.869319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.870382 (+ 1063us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.870587 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.871526 (+ 939us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.871629 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:50:37.873607 19688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: Bootstrap starting.
Sep 28 02:50:37 dist-test-slave-r238 krb5kdc[19456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563837, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:50:37.875571 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:37.876662 19688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:37.876873 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.866550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47204)
0928 02:50:37.866687 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.866702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.866780 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.867124 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.867128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.867319 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.867438 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.867447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.868937 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.868941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.869795 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.869824 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.870463 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.870467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.870563 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.870995 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.871008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.872757 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.875284 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.875287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.875290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.875523 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.875835 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.875838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.875840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.875928 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.876331 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.876334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.876428 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.876517 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.876702 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:50:37.877374 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.866587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.28.248.188:40417)
0928 02:50:37.866928 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.866934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.866945 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:50:37.866971 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.866973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.867003 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.867073 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.867541 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.868799 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.869969 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.870332 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.870780 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.872893 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.872897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.872899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.872922 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.875159 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.875639 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.875642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.875644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.875720 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.876037 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.876041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.876043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.876220 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.877037 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.877111 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.877215 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20260928 02:50:37.877228 19511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37218
I20260928 02:50:37.879245 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.869876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:40417 (local address 127.0.0.1:47218)
0928 02:50:37.869969 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:37.869985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:37.870058 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:37.870443 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:37.870446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:37.870616 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:37.870727 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.870737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.871751 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.871755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.872542 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:37.872552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.873311 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.873315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:37.873400 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.874100 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:37.874114 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:37.875699 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:37.877678 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.877681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.877683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.877922 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.878213 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:37.878216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:37.878218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:37.878302 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:37.878732 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:37.878735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:37.878833 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:50:37.878932 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:50:37.879086 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:50:37.879844 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.870100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.28.248.188:40417)
0928 02:50:37.870219 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.870223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.870235 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:50:37.870265 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.870267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.870308 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.870832 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.870912 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.871627 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.872894 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.873191 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.873295 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.875836 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.875840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.875842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.875864 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.877551 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.878041 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.878044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.878046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.878116 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.878408 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.878411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.878412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.878617 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.879523 (+ 906us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.879596 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.879702 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:50:37.879835 19537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: Bootstrap starting.
I20260928 02:50:37.882045 19537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:37.882580 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: Bootstrap starting.
I20260928 02:50:37.882942 19537 log.cc:824] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:37.883451 19688 log.cc:824] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:37.884970 19612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:37.885232 19537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: No bootstrap required, opened a new log
I20260928 02:50:37.885452 19688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: No bootstrap required, opened a new log
I20260928 02:50:37.885779 19612 log.cc:824] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:37.887650 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: No bootstrap required, opened a new log
I20260928 02:50:37.888254 19537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.888501 19537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb2c42a657c49a68c6cc7b675fee0aa, State: Initialized, Role: FOLLOWER
I20260928 02:50:37.888882 19537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.889426 19697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.889461 19688 raft_consensus.cc:348] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.889681 19688 raft_consensus.cc:729] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a70241bd90e04affbc2b56de39331b99, State: Initialized, Role: FOLLOWER
I20260928 02:50:37.889802 19697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.889966 19688 consensus_queue.cc:260] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.891290 19688 sys_catalog.cc:564] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:37.891320 19537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:37.891407 19511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:50:37.891563 19511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:37.891974 19612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.892161 19612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffc74c8fe504188bd10a99097fdfe69, State: Initialized, Role: FOLLOWER
I20260928 02:50:37.892424 19612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.892894 19511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
I20260928 02:50:37.893838 19511 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [CANDIDATE]: Term 1 election: Requested vote from peers 7ffc74c8fe504188bd10a99097fdfe69 (127.28.248.189:45439), a70241bd90e04affbc2b56de39331b99 (127.28.248.188:40417)
I20260928 02:50:37.894770 19662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a70241bd90e04affbc2b56de39331b99"
I20260928 02:50:37.894987 19662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:50:37.896751 19490 tablet.cc:2369] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:50:37.897238 19720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:37.897324 19720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:37.897356 19662 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb2c42a657c49a68c6cc7b675fee0aa in term 1.
I20260928 02:50:37.897594 19698 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.897702 19698 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.898211 19488 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb2c42a657c49a68c6cc7b675fee0aa, a70241bd90e04affbc2b56de39331b99; no voters:
I20260928 02:50:37.898504 19697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:37.898681 19697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [term 1 LEADER]: Becoming Leader. State: Replica: 6cb2c42a657c49a68c6cc7b675fee0aa, State: Running, Role: LEADER
I20260928 02:50:37.898986 19697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } }
W20260928 02:50:37.900676 19721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:37.900756 19721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:37.900668 19697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb2c42a657c49a68c6cc7b675fee0aa. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.900856 19697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:37.898197 19469 master_runner.cc:186] Error getting master registration for 127.28.248.189:45439: OK, instance_id { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" instance_seqno: 1790563837615267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:50:37.902140 19469 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:50:37.902463 19730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:37.902488 19612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:37.902541 19585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ffc74c8fe504188bd10a99097fdfe69"
I20260928 02:50:37.902741 19585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:37.903342 19730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:37.904261 19585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb2c42a657c49a68c6cc7b675fee0aa in term 1.
I20260928 02:50:37.901222 19619 master_runner.cc:186] Error getting master registration for 127.28.248.189:45439: OK, instance_id { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" instance_seqno: 1790563837615267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:50:37.904387 19619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:50:37.907853 19741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:37.907936 19741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:37.909900 19585 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [term 1 FOLLOWER]: Refusing update from remote peer 6cb2c42a657c49a68c6cc7b675fee0aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:37.911357 19662 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [term 1 FOLLOWER]: Refusing update from remote peer 6cb2c42a657c49a68c6cc7b675fee0aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:37.912042 19727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:37.912919 19727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:37.920193 19698 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb2c42a657c49a68c6cc7b675fee0aa. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.920375 19698 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.922894 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.923121 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.923250 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb2c42a657c49a68c6cc7b675fee0aa. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.923411 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.929348 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.910897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.28.248.188:40417)
0928 02:50:37.911009 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.911013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.920198 (+ 9185us) server_negotiation.cc:407] Connection header received
0928 02:50:37.920378 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.920382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.920439 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.920486 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.921530 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.922244 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.923377 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.923688 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.923764 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.925675 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.925678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.925681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.925707 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.927485 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.927993 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.927997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.927999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.928082 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.928375 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.928378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.928380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.928582 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.928715 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.929060 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.929175 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:50:37.933069 19727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.933228 19727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:37.933347 19742 mvcc.cc:205] Tried to move back new op lower bound from 7334149480066207744 to 7334149480037433344. Current Snapshot: MvccSnapshot[applied={T|T < 7334149480066207744}]
I20260928 02:50:37.934055 19697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.934175 19697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:37.934641 19698 sys_catalog.cc:455] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.934757 19698 sys_catalog.cc:458] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.935380 19750 mvcc.cc:205] Tried to move back new op lower bound from 7334149480066207744 to 7334149480037433344. Current Snapshot: MvccSnapshot[applied={T|T < 7334149480066207744}]
I20260928 02:50:37.935380 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb2c42a657c49a68c6cc7b675fee0aa" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 34631 } } peers { permanent_uuid: "7ffc74c8fe504188bd10a99097fdfe69" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 45439 } } peers { permanent_uuid: "a70241bd90e04affbc2b56de39331b99" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 40417 } } }
I20260928 02:50:37.935528 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:37.936058 19748 mvcc.cc:205] Tried to move back new op lower bound from 7334149480066207744 to 7334149480037433344. Current Snapshot: MvccSnapshot[applied={T|T < 7334149480066207744}]
I20260928 02:50:37.940754 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.910383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.28.248.190:34631)
0928 02:50:37.910645 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.910652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.918830 (+ 8178us) server_negotiation.cc:407] Connection header received
0928 02:50:37.919039 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.919043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.919095 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.919151 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.923675 (+ 4524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.924541 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.927066 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.927591 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.927679 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.936748 (+ 9069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.936753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.936756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.936787 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.938654 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.939286 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.939290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.939292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.939393 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.939723 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.939726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.939728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.939964 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.940073 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.940437 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.940564 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260928 02:50:37.941512 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:37.910497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.28.248.189:45439)
0928 02:50:37.910746 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:37.910752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:37.916949 (+ 6197us) server_negotiation.cc:407] Connection header received
0928 02:50:37.917162 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:37.917166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:37.917219 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:37.917276 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:37.921429 (+ 4153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.922316 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.925857 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:37.926233 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:37.926318 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:37.937771 (+ 11453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:37.937775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:37.937778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:37.937807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:37.939660 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.940171 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.940174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.940176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.940261 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:37.940537 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:37.940540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:37.940542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:37.940775 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:37.940911 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:37.941264 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:50:37.941380 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 02:50:37.943935 19730 catalog_manager.cc:1347] Generated new cluster ID: fb11537f74c341189c8af4f3cf047f26
I20260928 02:50:37.944020 19730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:37.972721 19730 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:37.974092 19730 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:37.988294 19730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb2c42a657c49a68c6cc7b675fee0aa: Generated new TSK 0
I20260928 02:50:37.989359 19730 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:50:38.021860 19691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.019286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.28.248.188:40417)
0928 02:50:38.019414 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.019420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.020692 (+ 1272us) server_negotiation.cc:407] Connection header received
0928 02:50:38.020901 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.020904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.020958 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.021013 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.021719 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47250: BlockingRecv error: recv got EOF from 127.0.0.1:47250 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 02:50:38.023372 19540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.020085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.28.248.190:34631)
0928 02:50:38.020197 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.020201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.022511 (+ 2310us) server_negotiation.cc:407] Connection header received
0928 02:50:38.022676 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.022679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.022726 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.022775 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.023254 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37236: BlockingRecv error: recv got EOF from 127.0.0.1:37236 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:50:38.025233 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.023957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.28.248.189:45439)
0928 02:50:38.024073 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.024079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.024439 (+ 360us) server_negotiation.cc:407] Connection header received
0928 02:50:38.024560 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.024563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.024635 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.024721 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.025117 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41394: BlockingRecv error: recv got EOF from 127.0.0.1:41394 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 02:50:38.030465 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.028980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.28.248.189:45439)
0928 02:50:38.029077 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.029081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.029558 (+ 477us) server_negotiation.cc:407] Connection header received
0928 02:50:38.029769 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.029772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.029850 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.029896 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.030355 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41400: BlockingRecv error: recv got EOF from 127.0.0.1:41400 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 02:50:38.035485 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.034075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.28.248.189:45439)
0928 02:50:38.034167 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.034171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.034621 (+ 450us) server_negotiation.cc:407] Connection header received
0928 02:50:38.034789 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.034792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.034837 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.034886 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.035376 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41412: BlockingRecv error: recv got EOF from 127.0.0.1:41412 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 02:50:38.040906 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.039562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.28.248.189:45439)
0928 02:50:38.039654 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.039658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.040073 (+ 415us) server_negotiation.cc:407] Connection header received
0928 02:50:38.040236 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.040239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.040283 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.040330 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.040811 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41424: BlockingRecv error: recv got EOF from 127.0.0.1:41424 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 02:50:38.048342 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.047001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.28.248.189:45439)
0928 02:50:38.047126 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.047130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.047558 (+ 428us) server_negotiation.cc:407] Connection header received
0928 02:50:38.047719 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.047722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.047765 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.047811 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.048249 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41432: BlockingRecv error: recv got EOF from 127.0.0.1:41432 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:50:38.060439 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.058875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.28.248.189:45439)
0928 02:50:38.058980 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.058985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.059462 (+ 477us) server_negotiation.cc:407] Connection header received
0928 02:50:38.059676 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.059680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.059738 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.059794 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.060334 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41436: BlockingRecv error: recv got EOF from 127.0.0.1:41436 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:50:38.081249 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.079357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.28.248.189:45439)
0928 02:50:38.079505 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.079510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.080104 (+ 594us) server_negotiation.cc:407] Connection header received
0928 02:50:38.080343 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.080347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.080413 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.080492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.081135 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41440: BlockingRecv error: recv got EOF from 127.0.0.1:41440 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:50:38.117209 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.115879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.28.248.189:45439)
0928 02:50:38.116005 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.116010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.116277 (+ 267us) server_negotiation.cc:407] Connection header received
0928 02:50:38.116446 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.116449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.116509 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.116600 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.117089 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41452: BlockingRecv error: recv got EOF from 127.0.0.1:41452 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 02:50:38.184789 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.183460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.28.248.189:45439)
0928 02:50:38.183656 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.183660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.183817 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:50:38.184027 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.184029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.184079 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.184151 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.184661 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41462: BlockingRecv error: recv got EOF from 127.0.0.1:41462 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 02:50:38.316197 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.314868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.28.248.189:45439)
0928 02:50:38.315025 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.315032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.315160 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:50:38.315287 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.315290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.315341 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.315414 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.316067 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41470: BlockingRecv error: recv got EOF from 127.0.0.1:41470 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 02:50:38.575754 19617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:38.574584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.28.248.189:45439)
0928 02:50:38.574713 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:38.574719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:38.574916 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:50:38.575027 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:38.575029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:38.575082 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:38.575156 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:38.575629 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41474: BlockingRecv error: recv got EOF from 127.0.0.1:41474 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:50:38.898773 19720 catalog_manager.cc:1259] Loaded cluster ID: fb11537f74c341189c8af4f3cf047f26
I20260928 02:50:38.898931 19720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: loading cluster ID for follower catalog manager: success
I20260928 02:50:38.900770 19720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: acquiring CA information for follower catalog manager: success
I20260928 02:50:38.902024 19720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a70241bd90e04affbc2b56de39331b99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:50:38.909157 19741 catalog_manager.cc:1259] Loaded cluster ID: fb11537f74c341189c8af4f3cf047f26
I20260928 02:50:38.909219 19741 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: loading cluster ID for follower catalog manager: success
I20260928 02:50:38.910763 19741 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: acquiring CA information for follower catalog manager: success
I20260928 02:50:38.911806 19741 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ffc74c8fe504188bd10a99097fdfe69: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:50:39.094209 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:39.089792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.28.248.189:45439)
0928 02:50:39.090047 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:39.090051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:39.090066 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:50:39.090199 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:39.090202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:39.090412 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:39.090533 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:39.090853 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.091496 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.092574 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.092878 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.092963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:39.093278 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:39.093794 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:39.093909 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:39.093956 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:50:39.094007 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260928 02:50:39.100943 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:39.097331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.28.248.188:40417)
0928 02:50:39.097574 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:39.097577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:39.097590 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:39.097663 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:39.097665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:39.097774 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:39.097868 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:39.098069 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.098599 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.099404 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.099616 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.099688 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:39.100060 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:39.100480 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:39.100571 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:39.100711 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:50:39.100783 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260928 02:50:39.108209 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:39.103805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.28.248.188:40417)
0928 02:50:39.103912 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:39.103919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:39.104383 (+ 464us) server_negotiation.cc:407] Connection header received
0928 02:50:39.104587 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:39.104592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:39.104690 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:39.104765 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:39.105024 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.105666 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.106812 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.107141 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.107215 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:39.107494 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:39.107569 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:39.107646 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:39.108010 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:50:39.108073 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:50:39.108317 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:39.103732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.28.248.189:45439)
0928 02:50:39.104180 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:39.104184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:39.104681 (+ 497us) server_negotiation.cc:407] Connection header received
0928 02:50:39.104886 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:39.104889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:39.104967 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:39.105023 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:39.105396 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.106134 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.107065 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.107370 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.107470 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:39.107757 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:39.107869 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:39.107942 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:39.108109 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:50:39.108175 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260928 02:50:39.109594 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:39.103955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.28.248.190:34631)
0928 02:50:39.104338 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:39.104345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:39.104363 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:50:39.104471 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:39.104475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:39.104693 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:39.104807 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:39.105155 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.105949 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.107543 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:39.107892 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:39.107987 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:39.108708 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:39.109000 (+ 292us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:39.109112 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:39.109310 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:50:39.109485 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260928 02:50:39.113072 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19469
I20260928 02:50:39.119810 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19544
I20260928 02:50:39.125762 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19619
2026-09-28T02:50:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3107 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:50:39 dist-test-slave-r238 krb5kdc[19838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:50:39 dist-test-slave-r238 krb5kdc[19838](info): set up 2 sockets
Sep 28 02:50:39 dist-test-slave-r238 krb5kdc[19838](info): commencing operation
krb5kdc: starting...
W20260928 02:50:40.177062 29666 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/test-user.keytab.
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.248.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
Entry for principal kudu/127.28.248.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab.
I20260928 02:50:40.301759 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:33279
--webserver_interface=127.28.248.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:33279,127.28.248.189:44411,127.28.248.188:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:50:40.412371 19851 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:50:40.412719 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:40.412784 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:40.412860 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:40.417048 19851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:40.417135 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:40.417171 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:40.417209 19851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:40.417246 19851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:40.422731 19851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:33279,127.28.248.189:44411,127.28.248.188:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.190.keytab
--principal=kudu/127.28.248.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.190:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.248.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:40.423805 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:40.424849 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:40.433162 19856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:40.433162 19859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:40.433156 19857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:40.433425 19851 server_base.cc:1034] running on GCE node
I20260928 02:50:40.434054 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:40.435298 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:40.436501 19851 hybrid_clock.cc:648] HybridClock initialized: now 1790563840436489 us; error 29 us; skew 500 ppm
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:40.439368 19851 init.cc:377] Logged in from keytab as kudu/127.28.248.190@KRBTEST.COM (short username kudu)
I20260928 02:50:40.440253 19851 webserver.cc:460] Webserver started at http://127.28.248.190:39599/ using document root <none> and password file <none>
I20260928 02:50:40.440889 19851 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:40.440979 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:40.441210 19851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:40.443230 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data/instance:
uuid: "a39aa28045cb4ff3ad81c12b3147dcf1"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "30b6f18f24791a6679cd813bccbdf3d0"
server_key_iv: "a1e96a4577f03ac927a9e38cd0102250"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.443779 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal/instance:
uuid: "a39aa28045cb4ff3ad81c12b3147dcf1"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "30b6f18f24791a6679cd813bccbdf3d0"
server_key_iv: "a1e96a4577f03ac927a9e38cd0102250"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.447377 19851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 02:50:40.449798 19866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:40.451021 19851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:50:40.451155 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal
uuid: "a39aa28045cb4ff3ad81c12b3147dcf1"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "30b6f18f24791a6679cd813bccbdf3d0"
server_key_iv: "a1e96a4577f03ac927a9e38cd0102250"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.451260 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:40.463685 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:40.466820 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:40.467011 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:40.474103 19851 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.190:33279
I20260928 02:50:40.474396 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.190:33279 every 8 connection(s)
I20260928 02:50:40.475050 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/data/info.pb
I20260928 02:50:40.477710 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19851
I20260928 02:50:40.477773 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:40.477865 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-0/wal/instance
I20260928 02:50:40.478142 29666 external_mini_cluster.cc:1428] Setting key 1a9cdba50e53304c53e7ab11e697d9fa
I20260928 02:50:40.483018 19919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.248.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.189@KRBTEST.COM" created.
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:40.499568 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.485981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:33279 (local address 127.0.0.1:36066)
0928 02:50:40.486627 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.486674 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.487505 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.488218 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.488222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.488546 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.488966 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.488982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.490553 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.490556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.491405 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.491414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.492157 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.492161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.492261 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.492949 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.492983 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.494575 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.496671 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.496678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.496687 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.496928 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.497257 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.497262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.497264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.497374 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.497935 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.497941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.498077 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.498241 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.498625 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":107,"spinlock_wait_cycles":52608,"thread_start_us":150,"threads_started":1}
I20260928 02:50:40.499568 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.485981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.28.248.190:33279)
0928 02:50:40.486498 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.486516 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.486807 (+ 291us) server_negotiation.cc:407] Connection header received
0928 02:50:40.487829 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.487833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.488034 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.488192 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.489132 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.490417 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.491589 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.491996 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.492176 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.494744 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.494749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.494760 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.494780 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.496521 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.497050 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.497055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.497059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.497160 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.497471 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.497474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.497475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.497819 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.497962 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.498313 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.498550 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":150,"threads_started":1}
I20260928 02:50:40.502085 19919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:40.502777 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.502530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54740)
0928 02:50:40.502646 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.502719 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:50:40.502962 19869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
I20260928 02:50:40.503856 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.503638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54746)
0928 02:50:40.503742 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.503787 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:50:40.504011 19919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:44411: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab.
Entry for principal kudu/127.28.248.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab.
I20260928 02:50:40.507484 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:44411
--webserver_interface=127.28.248.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab
--principal=kudu/127.28.248.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:33279,127.28.248.189:44411,127.28.248.188:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:50:40.553306 19919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } attempt: 1
I20260928 02:50:40.554272 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.553943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54754)
0928 02:50:40.554109 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.554190 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:50:40.555150 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.554906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54768)
0928 02:50:40.555027 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.555087 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:50:40.555368 19919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:44411: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
W20260928 02:50:40.616297 19926 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:50:40.616565 19926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:40.616588 19926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:40.616700 19926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:50:40.620639 19926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:40.620687 19926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:40.620705 19926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:40.620721 19926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:40.620736 19926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:40.623553 19919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } attempt: 2
I20260928 02:50:40.624433 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.624087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54784)
0928 02:50:40.624264 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.624354 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:50:40.625265 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.625040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54794)
0928 02:50:40.625150 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.625201 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:50:40.625439 19919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.189:44411: Network error: Client connection negotiation failed: client connection to 127.28.248.189:44411: connect: Connection refused (error 111)
I20260928 02:50:40.626344 19926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:33279,127.28.248.189:44411,127.28.248.188:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.189.keytab
--principal=kudu/127.28.248.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.189:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.248.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:40.627259 19926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:40.628170 19926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:40.636950 19931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:40.637234 19926 server_base.cc:1034] running on GCE node
W20260928 02:50:40.637044 19934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:40.636950 19932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:40.637957 19926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:40.639004 19926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:40.640183 19926 hybrid_clock.cc:648] HybridClock initialized: now 1790563840640168 us; error 23 us; skew 500 ppm
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:40.643018 19926 init.cc:377] Logged in from keytab as kudu/127.28.248.189@KRBTEST.COM (short username kudu)
I20260928 02:50:40.643900 19926 webserver.cc:460] Webserver started at http://127.28.248.189:36985/ using document root <none> and password file <none>
I20260928 02:50:40.644487 19926 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:40.644586 19926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:40.644830 19926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:40.646936 19926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data/instance:
uuid: "199f8584466e43cd87b4a27a7ce37a60"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "0d0c393a42e0cc17ca14380079271780"
server_key_iv: "a970cbe06fe80c80749fdb6db9f75d13"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.647559 19926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal/instance:
uuid: "199f8584466e43cd87b4a27a7ce37a60"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "0d0c393a42e0cc17ca14380079271780"
server_key_iv: "a970cbe06fe80c80749fdb6db9f75d13"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.651441 19926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 02:50:40.654158 19941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:40.655603 19926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:50:40.655747 19926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal
uuid: "199f8584466e43cd87b4a27a7ce37a60"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "0d0c393a42e0cc17ca14380079271780"
server_key_iv: "a970cbe06fe80c80749fdb6db9f75d13"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.655861 19926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:40.671247 19926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:40.674578 19926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:40.674829 19926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:40.682546 19926 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.189:44411
I20260928 02:50:40.682849 19993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.189:44411 every 8 connection(s)
I20260928 02:50:40.683590 19926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/data/info.pb
I20260928 02:50:40.686456 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:40.690112 19994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:40.693717 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 19926
I20260928 02:50:40.693928 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-1/wal/instance
I20260928 02:50:40.694232 29666 external_mini_cluster.cc:1428] Setting key 2726131068cae63de03e122a530d3daa
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.248.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.248.188@KRBTEST.COM" created.
I20260928 02:50:40.705265 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.691950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.28.248.190:33279)
0928 02:50:40.692104 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.692108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.693077 (+ 969us) server_negotiation.cc:407] Connection header received
0928 02:50:40.694241 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.694245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.694309 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.694421 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.695634 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.696254 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.698075 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.698440 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.698536 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.701597 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.701606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.701608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.701631 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.703105 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.703670 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.703680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.703682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.703753 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.704093 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.704096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.704098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.704273 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.704385 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.704960 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.705092 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:50:40.706229 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.692330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:33279 (local address 127.0.0.1:36074)
0928 02:50:40.692910 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.692955 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.694017 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.694593 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.694600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.694976 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.695435 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.695447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.696399 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.696407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.697922 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.697938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.698618 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.698622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.698850 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.699576 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.699603 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.701415 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.703233 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.703238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.703253 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.703561 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.703851 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.703854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.703856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.703993 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.704384 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.704391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.704710 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.704896 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.705218 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":132,"threads_started":1}
I20260928 02:50:40.707756 19994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
I20260928 02:50:40.716661 19919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } attempt: 3
I20260928 02:50:40.717060 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.708297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54806)
0928 02:50:40.708509 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.708520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.708612 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.709070 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.709073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.709240 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.709319 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.709324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.710256 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.710259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.710792 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.710799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.711369 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.711372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.711459 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.711929 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.711942 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.713134 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.714993 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.714998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.715000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.715278 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.715607 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.715610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.715612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.715742 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.716326 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.716328 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.716414 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.716512 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.716791 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260928 02:50:40.717065 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.708423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.28.248.189:44411)
0928 02:50:40.708740 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.708750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.708777 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:50:40.708806 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.708810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.708951 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.709093 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.709420 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.710133 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.710900 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.711260 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.711374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.713269 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.713277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.713290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.713309 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.714815 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.715391 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.715395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.715396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.715468 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.715879 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.715882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.715883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.716212 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.716337 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.716701 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.716872 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260928 02:50:40.719884 19994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:40.720894 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.720392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45442)
0928 02:50:40.720749 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.720825 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
W20260928 02:50:40.721657 19943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
I20260928 02:50:40.722623 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.722362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45448)
0928 02:50:40.722475 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.722555 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.248.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab.
Entry for principal kudu/127.28.248.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab.
W20260928 02:50:40.723176 19994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:34877: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
I20260928 02:50:40.723816 29666 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9u0ndH/build/debug/bin/kudu
/tmp/dist-test-task9u0ndH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:34877
--webserver_interface=127.28.248.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab
--principal=kudu/127.28.248.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.248.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.248.190:33279,127.28.248.189:44411,127.28.248.188:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
I20260928 02:50:40.729180 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.717218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54812)
0928 02:50:40.717359 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.717377 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.717476 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.718080 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.718083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.718320 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.718435 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.718441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.719875 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.719878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.720888 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.720900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.721737 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.721741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.721920 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.722474 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.722487 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.725807 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.727598 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.727602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.727604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.727874 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.728160 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.728162 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.728164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.728255 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.728663 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.728666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.728770 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.728881 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.729025 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:50:40.729198 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.717252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.28.248.189:44411)
0928 02:50:40.717655 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.717664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.717796 (+ 132us) server_negotiation.cc:407] Connection header received
0928 02:50:40.717840 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.717843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.717905 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.718597 (+ 692us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.718746 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.719690 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.721143 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.721615 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.721715 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.725978 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.725981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.725983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.726003 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.727465 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.727977 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.727982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.727984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.728065 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.728351 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.728354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.728356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.728540 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.728666 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.728932 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.729037 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":130,"threads_started":1}
I20260928 02:50:40.729946 19919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:40.731037 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.730797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45456)
0928 02:50:40.730921 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.730973 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:50:40.731765 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.731535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45466)
0928 02:50:40.731680 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.731721 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 02:50:40.731936 19919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:34877: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
I20260928 02:50:40.763106 19919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } attempt: 1
I20260928 02:50:40.764010 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.763693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45480)
0928 02:50:40.763858 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.763926 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:50:40.764768 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.764525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45494)
0928 02:50:40.764670 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.764717 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 02:50:40.764973 19919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:34877: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
I20260928 02:50:40.772660 19994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } attempt: 1
I20260928 02:50:40.773443 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.773179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45510)
0928 02:50:40.773310 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.773375 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:50:40.774188 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.773981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45518)
0928 02:50:40.774086 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.774138 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:50:40.774380 19994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:34877: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
W20260928 02:50:40.836714 20002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:50:40.836963 20002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:50:40.837023 20002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:50:40.837126 20002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:50:40.840138 19919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } attempt: 2
I20260928 02:50:40.841099 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.840804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45528)
0928 02:50:40.840953 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.841021 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:50:40.841409 20002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:50:40.841483 20002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:50:40.841523 20002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:50:40.841555 20002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:50:40.841586 20002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:50:40.841799 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.841621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45532)
0928 02:50:40.841710 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.841751 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 02:50:40.841933 19919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:34877: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
I20260928 02:50:40.842553 19994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } attempt: 2
I20260928 02:50:40.844251 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.843918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45542)
0928 02:50:40.844091 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.844171 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:50:40.844975 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.844727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45556)
0928 02:50:40.844840 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.844898 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 02:50:40.845119 19994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.248.188:34877: Network error: Client connection negotiation failed: client connection to 127.28.248.188:34877: connect: Connection refused (error 111)
I20260928 02:50:40.848204 20002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.248.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.248.190:33279,127.28.248.189:44411,127.28.248.188:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/krb5kdc/kudu_127.28.248.188.keytab
--principal=kudu/127.28.248.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.248.188:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.248.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 02:07:03 UTC on ccycloud.quasar-wlajti.root.comops.site
build id 4676524
I20260928 02:50:40.849339 20002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:50:40.850350 20002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:50:40.858690 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:40.858790 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:50:40.858790 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:50:40.858989 20002 server_base.cc:1034] running on GCE node
I20260928 02:50:40.859694 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:50:40.860970 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:50:40.862150 20002 hybrid_clock.cc:648] HybridClock initialized: now 1790563840862117 us; error 47 us; skew 500 ppm
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:50:40.864887 20002 init.cc:377] Logged in from keytab as kudu/127.28.248.188@KRBTEST.COM (short username kudu)
I20260928 02:50:40.865736 20002 webserver.cc:460] Webserver started at http://127.28.248.188:35593/ using document root <none> and password file <none>
I20260928 02:50:40.866302 20002 fs_manager.cc:359] Metadata directory not provided
I20260928 02:50:40.866393 20002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:50:40.866628 20002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:50:40.868745 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data/instance:
uuid: "0c388365249e4cda99f4f50cefe0ab4f"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "311b775f69421aa14d03d68774d7abc2"
server_key_iv: "1ff67ee8a7ef27bf43dae7ee7ccd6466"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.869321 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal/instance:
uuid: "0c388365249e4cda99f4f50cefe0ab4f"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "311b775f69421aa14d03d68774d7abc2"
server_key_iv: "1ff67ee8a7ef27bf43dae7ee7ccd6466"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.873183 20002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:50:40.875600 20017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:50:40.877297 20002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:50:40.877403 20002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data,/tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal
uuid: "0c388365249e4cda99f4f50cefe0ab4f"
format_stamp: "Formatted at 2026-09-28 02:50:40 on dist-test-slave-r238"
server_key: "311b775f69421aa14d03d68774d7abc2"
server_key_iv: "1ff67ee8a7ef27bf43dae7ee7ccd6466"
server_key_version: "encryptionkey@0"
I20260928 02:50:40.877489 20002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:50:40.896320 20002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:50:40.899618 20002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:50:40.899794 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:50:40.907140 20002 rpc_server.cc:307] RPC server started. Bound to: 127.28.248.188:34877
I20260928 02:50:40.907423 20069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.248.188:34877 every 8 connection(s)
I20260928 02:50:40.908193 20002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/data/info.pb
I20260928 02:50:40.910661 29666 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9u0ndH/build/debug/bin/kudu as pid 20002
I20260928 02:50:40.910794 29666 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9u0ndH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563560542449-29666-0/minicluster-data/master-2/wal/instance
I20260928 02:50:40.911108 29666 external_mini_cluster.cc:1428] Setting key 1b315d754368308b6729fcad5efd81e8
I20260928 02:50:40.911280 20070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:50:40.916347 20070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:40.926671 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.912441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.28.248.188:34877)
0928 02:50:40.912874 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.912893 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.912937 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:50:40.914042 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.914066 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.914510 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.914883 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.915560 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.917058 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.917991 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.918628 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.918784 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.921500 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.921506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.921518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.921547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.923508 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.923968 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.923974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.923982 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.924092 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.924414 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.924417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.924437 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.925026 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.925240 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.925469 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.925745 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":147,"threads_started":1}
I20260928 02:50:40.927284 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.918178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.190:33279 (local address 127.0.0.1:36084)
0928 02:50:40.918542 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.918592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.918715 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.919020 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.919026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.919380 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.919497 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.919503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.920212 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.920215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.921186 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.921196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.921729 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.921731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.921854 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.922377 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.922398 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.923907 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.925769 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.925776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.925780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.925995 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.926273 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.926278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.926285 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.926357 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.926768 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.926776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.926913 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.927029 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.927153 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260928 02:50:40.927384 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.918280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.28.248.190:33279)
0928 02:50:40.918421 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.918426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.918680 (+ 254us) server_negotiation.cc:407] Connection header received
0928 02:50:40.918807 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.918811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.918872 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.918970 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.919618 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.920096 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.921312 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.921610 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.921726 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.924037 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.924041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.924044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.924067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.925654 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.926088 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.926092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.926094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.926176 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.926444 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.926447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.926448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.926645 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.926749 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.927123 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.927235 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:50:40.928571 20070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:40.936275 19994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } attempt: 3
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.190@KRBTEST.COM
I20260928 02:50:40.941099 19919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } attempt: 3
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.189@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:50:40.944031 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.929333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.28.248.189:44411)
0928 02:50:40.929486 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.929492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.929508 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:50:40.929543 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.929546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.929584 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.929671 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.930061 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.930571 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.931307 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.931575 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.931690 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.939954 (+ 8264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.939960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.939966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.940001 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.941730 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.942270 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.942276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.942280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.942365 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.942723 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.942727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.942729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.943074 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.943192 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.943664 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.943806 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:50:40.944819 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.929060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.189:44411 (local address 127.0.0.1:54828)
0928 02:50:40.929304 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.929316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.929404 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.929684 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.929686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.929848 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.929932 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.929937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.930677 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.930684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.931191 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.931198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.931687 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.931691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.931776 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.937749 (+ 5973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.937767 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.939796 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.941874 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.941880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.941885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.942156 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.942475 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.942481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.942485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.942598 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.943275 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.943278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.943458 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.944475 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.944654 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 02:50:40.944887 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.929065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.28.248.190:33279)
0928 02:50:40.929161 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.929166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.929184 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:50:40.929310 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.929313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.929356 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.929437 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.930022 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.930495 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.931239 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.931460 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.931579 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.940141 (+ 8562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.940147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.940150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.940187 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.942709 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.943202 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.943208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.943213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.943302 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.943746 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.943751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.943756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.943972 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.944194 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.944593 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.944747 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:50:40.945945 20070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:50:40.946112 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.936851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45570)
0928 02:50:40.937038 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.937054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.937182 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.937483 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.937487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.937698 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.937830 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.937837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.938778 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.938783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.939374 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.939383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.940014 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.940017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.940104 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.940710 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.940728 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.942546 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.944574 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.944578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.944580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.944833 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.945127 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.945130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.945132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.945217 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.945663 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.945666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.945749 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.945820 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.945939 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:50:40.946609 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.936932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.28.248.188:34877)
0928 02:50:40.937055 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.937060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.937125 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:50:40.937295 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.937298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.937353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.937467 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.937966 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.938661 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.939495 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.939848 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.940007 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.942670 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.942675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.942677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.942703 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.944441 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.944944 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.944947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.944949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.945027 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.945327 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.945330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.945332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.945551 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.946274 (+ 723us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.946344 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.946450 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.190@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:50:40.950284 19994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: Bootstrap starting.
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.248.188@KRBTEST.COM for kudu/127.28.248.188@KRBTEST.COM
I20260928 02:50:40.952889 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.941820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45582)
0928 02:50:40.942036 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.942053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.942133 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.943130 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.943136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.943332 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.943465 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.943475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.945125 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.945128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.945931 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.945941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.946687 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.946693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.946786 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.947635 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.947649 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.949110 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.951119 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.951123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.951125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.951397 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.951784 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.951789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.951793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.951878 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.952344 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.952347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.952455 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.952546 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.952693 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
Sep 28 02:50:40 dist-test-slave-r238 krb5kdc[19838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.248.189@KRBTEST.COM
I20260928 02:50:40.953295 19994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:40.953557 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.941861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.28.248.188:34877)
0928 02:50:40.942846 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.942857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.942873 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:50:40.942912 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.942917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.942975 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.943582 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.943694 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.945001 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.946076 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.946551 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.947009 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.949238 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.949242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.949245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.949267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.951000 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.951518 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.951536 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.951538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.951665 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.952009 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.952013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.952017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.952233 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.953183 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.953258 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.953380 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"spinlock_wait_cycles":768,"thread_start_us":150,"threads_started":1}
I20260928 02:50:40.954231 19994 log.cc:824] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:40.956808 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1: Bootstrap starting.
I20260928 02:50:40.956926 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.946991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.28.248.188:34877)
0928 02:50:40.947458 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.947474 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.947488 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:50:40.947521 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.947526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.947570 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.947640 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.948174 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.948935 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.950371 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.950724 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.950828 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.953731 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.953734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.953736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.953759 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.955284 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.955793 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.955797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.955799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.955876 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.956131 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.956133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.956135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.956303 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.956384 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.956687 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.956785 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260928 02:50:40.957266 19994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: No bootstrap required, opened a new log
I20260928 02:50:40.957285 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.946324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.248.188:34877 (local address 127.0.0.1:45594)
0928 02:50:40.947090 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 02:50:40.947102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:50:40.947178 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:50:40.947741 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:50:40.947744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:50:40.947923 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:50:40.948050 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.948059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.949052 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.949057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.950241 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:50:40.950254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.950932 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.950935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:50:40.951035 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.951476 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:50:40.951538 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:50:40.953607 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:50:40.955392 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.955394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.955396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.955635 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.955966 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:50:40.955968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:50:40.955970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:50:40.956045 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:50:40.956414 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:50:40.956417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:50:40.956499 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 02:50:40.956593 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:50:40.957136 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":110,"threads_started":1}
I20260928 02:50:40.957911 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.946546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.28.248.189:44411)
0928 02:50:40.946665 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.946670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.946885 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:50:40.947212 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.947215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.947257 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.947336 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.948086 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.948806 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.950009 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.950297 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.950384 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.953864 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.953868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.953870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.953936 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.956059 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.956547 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.956550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.956552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.956655 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.956973 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.956977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.956978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.957167 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.957311 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.957679 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.957784 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:50:40.959762 19919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:40.959946 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: Bootstrap starting.
I20260928 02:50:40.960760 19919 log.cc:824] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:40.961524 19994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
I20260928 02:50:40.961838 19994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199f8584466e43cd87b4a27a7ce37a60, State: Initialized, Role: FOLLOWER
I20260928 02:50:40.962232 19994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
I20260928 02:50:40.962715 20070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: Neither blocks nor log segments found. Creating new log.
I20260928 02:50:40.962831 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1: No bootstrap required, opened a new log
I20260928 02:50:40.962837 29666 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:50:40.963620 20070 log.cc:824] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:50:40.963663 19994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:40.964051 19893 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36092
I20260928 02:50:40.965638 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: No bootstrap required, opened a new log
I20260928 02:50:40.967800 19919 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
I20260928 02:50:40.968169 19919 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39aa28045cb4ff3ad81c12b3147dcf1, State: Initialized, Role: FOLLOWER
I20260928 02:50:40.968743 19919 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
W20260928 02:50:40.969236 20091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:40.969303 20091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:40.967434 19926 master_runner.cc:186] Error getting master registration for 127.28.248.190:33279: OK, instance_id { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" instance_seqno: 1790563840472172 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:50:40.970297 19926 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:50:40.970466 20079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:40.970546 20079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:40.970485 20070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
I20260928 02:50:40.970721 20070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c388365249e4cda99f4f50cefe0ab4f, State: Initialized, Role: FOLLOWER
I20260928 02:50:40.971016 20070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
I20260928 02:50:40.971094 19919 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:40.971194 19893 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:50:40.971340 19893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:40.972281 20070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:50:40.972669 19893 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
I20260928 02:50:40.973435 20093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:40.974107 20093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:40.974112 20044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c388365249e4cda99f4f50cefe0ab4f"
I20260928 02:50:40.974112 19968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "199f8584466e43cd87b4a27a7ce37a60"
I20260928 02:50:40.974305 20044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:40.974305 19968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:50:40.975533 19968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39aa28045cb4ff3ad81c12b3147dcf1 in term 1.
I20260928 02:50:40.975533 20044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39aa28045cb4ff3ad81c12b3147dcf1 in term 1.
I20260928 02:50:40.976084 20094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:40.976217 20094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:40.977227 19869 leader_election.cc:302] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 199f8584466e43cd87b4a27a7ce37a60, a39aa28045cb4ff3ad81c12b3147dcf1; no voters:
I20260928 02:50:40.977499 20093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:50:40.977751 20093 raft_consensus.cc:686] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [term 1 LEADER]: Becoming Leader. State: Replica: a39aa28045cb4ff3ad81c12b3147dcf1, State: Running, Role: LEADER
I20260928 02:50:40.978094 20093 consensus_queue.cc:237] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } }
W20260928 02:50:40.979497 20116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:40.979580 20116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:40.979933 20093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39aa28045cb4ff3ad81c12b3147dcf1. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:40.980116 20093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: This master's current role is: LEADER
W20260928 02:50:40.983116 20109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:50:40.983228 20109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:50:40.983282 19851 master_runner.cc:263] No leader master found from master a39aa28045cb4ff3ad81c12b3147dcf1
I20260928 02:50:40.983352 19851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:50:40.983475 19893 leader_election.cc:288] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [CANDIDATE]: Term 1 election: Requested vote from peers 199f8584466e43cd87b4a27a7ce37a60 (127.28.248.189:44411), 0c388365249e4cda99f4f50cefe0ab4f (127.28.248.188:34877)
I20260928 02:50:40.984378 20002 master_runner.cc:263] No leader master found from master 0c388365249e4cda99f4f50cefe0ab4f
I20260928 02:50:40.984440 20002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:50:40.993744 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:40.986001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.28.248.189:44411)
0928 02:50:40.986121 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:40.986125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:40.986353 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:50:40.986515 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:40.986518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:40.986560 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:40.986651 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:40.987344 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.987849 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.988712 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:40.988949 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:40.989062 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:40.990502 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:40.990506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:40.990508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:40.990533 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:40.992036 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.992469 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.992474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.992476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.992557 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:40.992870 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:40.992873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:40.992875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:40.993065 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:40.993197 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:40.993442 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:50:40.993557 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:50:40.997118 20120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:50:40.998061 20120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:50:41.003834 20044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [term 1 FOLLOWER]: Refusing update from remote peer a39aa28045cb4ff3ad81c12b3147dcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:41.004710 20117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:41.008800 20094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39aa28045cb4ff3ad81c12b3147dcf1. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:41.009166 20094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:41.018877 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:41.008987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.28.248.190:33279)
0928 02:50:41.009105 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.009109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.009579 (+ 470us) server_negotiation.cc:407] Connection header received
0928 02:50:41.009621 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.009623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.009675 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.009745 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.010674 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:41.011465 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:41.012659 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:41.012994 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:41.013085 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:41.014841 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:41.014845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:41.014848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:41.014875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:41.016661 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:41.017173 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:41.017177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:41.017178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:41.017258 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:41.017546 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:41.017549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:41.017551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:41.017750 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:41.017877 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:41.018514 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 02:50:41.018686 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:50:41.029201 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:41.009589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.28.248.188:34877)
0928 02:50:41.009695 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.009700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.020350 (+ 10650us) server_negotiation.cc:407] Connection header received
0928 02:50:41.020537 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.020540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.020590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.020661 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.021480 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:41.022255 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:41.023317 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:41.023688 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:41.023757 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:41.025414 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:50:41.025418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:50:41.025421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:50:41.025447 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:50:41.027273 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:41.027772 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:41.027776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:41.027778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:41.027869 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:50:41.028150 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:50:41.028153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:50:41.028155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:50:41.028349 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:50:41.028474 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:41.028914 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 02:50:41.029036 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:50:41.032106 19968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [term 1 FOLLOWER]: Refusing update from remote peer a39aa28045cb4ff3ad81c12b3147dcf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:50:41.032821 20117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:50:41.037701 20079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39aa28045cb4ff3ad81c12b3147dcf1. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:41.037829 20079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:41.046518 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:41.046666 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:41.047930 20079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:41.048038 20079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:41.048513 20127 mvcc.cc:205] Tried to move back new op lower bound from 7334149492743258112 to 7334149492649820160. Current Snapshot: MvccSnapshot[applied={T|T < 7334149492743258112}]
I20260928 02:50:41.048965 20131 mvcc.cc:205] Tried to move back new op lower bound from 7334149492743258112 to 7334149492649820160. Current Snapshot: MvccSnapshot[applied={T|T < 7334149492743258112}]
I20260928 02:50:41.051894 20093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:41.052003 20093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:50:41.053932 20094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39aa28045cb4ff3ad81c12b3147dcf1" member_type: VOTER last_known_addr { host: "127.28.248.190" port: 33279 } } peers { permanent_uuid: "199f8584466e43cd87b4a27a7ce37a60" member_type: VOTER last_known_addr { host: "127.28.248.189" port: 44411 } } peers { permanent_uuid: "0c388365249e4cda99f4f50cefe0ab4f" member_type: VOTER last_known_addr { host: "127.28.248.188" port: 34877 } } }
I20260928 02:50:41.054039 20094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:50:41.055269 20120 catalog_manager.cc:1347] Generated new cluster ID: a0c9bb4b881d4170b25329c9cb60a629
I20260928 02:50:41.055335 20120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:50:41.055634 20128 mvcc.cc:205] Tried to move back new op lower bound from 7334149492743258112 to 7334149492649820160. Current Snapshot: MvccSnapshot[applied={T|T < 7334149492743258112}]
I20260928 02:50:41.086018 20120 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:50:41.087347 20120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:50:41.102794 20120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a39aa28045cb4ff3ad81c12b3147dcf1: Generated new TSK 0
I20260928 02:50:41.103693 20120 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:50:41.129040 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.125986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.28.248.189:44411)
0928 02:50:41.126108 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.126112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.127983 (+ 1871us) server_negotiation.cc:407] Connection header received
0928 02:50:41.128176 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.128179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.128228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.128281 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.128907 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54844: BlockingRecv error: recv got EOF from 127.0.0.1:54844 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:50:41.130925 19922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.126746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.28.248.190:33279)
0928 02:50:41.126874 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.126878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.129523 (+ 2645us) server_negotiation.cc:407] Connection header received
0928 02:50:41.129981 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.129985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.130038 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.130088 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.130792 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36112: BlockingRecv error: recv got EOF from 127.0.0.1:36112 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 02:50:41.131675 20075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.127344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.28.248.188:34877)
0928 02:50:41.127456 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.127460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.129719 (+ 2259us) server_negotiation.cc:407] Connection header received
0928 02:50:41.130216 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.130219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.130268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.130316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.131563 (+ 1247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45622: BlockingRecv error: recv got EOF from 127.0.0.1:45622 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 02:50:41.135958 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.134545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.28.248.189:44411)
0928 02:50:41.134643 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.134647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.135080 (+ 433us) server_negotiation.cc:407] Connection header received
0928 02:50:41.135278 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.135281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.135351 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.135401 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.135848 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54848: BlockingRecv error: recv got EOF from 127.0.0.1:54848 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 02:50:41.140727 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.139317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.28.248.189:44411)
0928 02:50:41.139414 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.139417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.139886 (+ 469us) server_negotiation.cc:407] Connection header received
0928 02:50:41.140058 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.140061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.140107 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.140154 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.140638 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54862: BlockingRecv error: recv got EOF from 127.0.0.1:54862 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 02:50:41.165815 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.157023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.28.248.189:44411)
0928 02:50:41.157152 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.157156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.164535 (+ 7379us) server_negotiation.cc:407] Connection header received
0928 02:50:41.164797 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.164801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.164878 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.164932 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.165685 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54866: BlockingRecv error: recv got EOF from 127.0.0.1:54866 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:50:41.174868 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.172984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.28.248.189:44411)
0928 02:50:41.173137 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.173142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.173809 (+ 667us) server_negotiation.cc:407] Connection header received
0928 02:50:41.174035 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.174039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.174096 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.174155 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.174750 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54882: BlockingRecv error: recv got EOF from 127.0.0.1:54882 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:50:41.187283 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.185558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.28.248.189:44411)
0928 02:50:41.185676 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.185681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.186280 (+ 599us) server_negotiation.cc:407] Connection header received
0928 02:50:41.186508 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.186511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.186568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.186622 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.187167 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54884: BlockingRecv error: recv got EOF from 127.0.0.1:54884 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:50:41.207407 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.206092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.28.248.189:44411)
0928 02:50:41.206214 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.206219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.206499 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:50:41.206634 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.206637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.206701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.206819 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.207284 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54898: BlockingRecv error: recv got EOF from 127.0.0.1:54898 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:50:41.242758 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.241630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.28.248.189:44411)
0928 02:50:41.241774 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.241778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.241971 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:50:41.242099 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.242102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.242151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.242235 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.242657 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54910: BlockingRecv error: recv got EOF from 127.0.0.1:54910 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:50:41.310483 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.309160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.28.248.189:44411)
0928 02:50:41.309305 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.309310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.309500 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:50:41.309690 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.309712 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.309784 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.309883 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.310363 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54922: BlockingRecv error: recv got EOF from 127.0.0.1:54922 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 02:50:41.442600 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.441395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.28.248.189:44411)
0928 02:50:41.441593 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.441597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.441716 (+ 119us) server_negotiation.cc:407] Connection header received
0928 02:50:41.441855 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.441857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.441904 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.441978 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.442469 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54926: BlockingRecv error: recv got EOF from 127.0.0.1:54926 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260928 02:50:41.701762 20000 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:50:41.700661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.28.248.189:44411)
0928 02:50:41.700811 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:41.700817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:41.700939 (+ 122us) server_negotiation.cc:407] Connection header received
0928 02:50:41.701049 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:41.701052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:41.701104 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:41.701187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:50:41.701651 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54932: BlockingRecv error: recv got EOF from 127.0.0.1:54932 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:50:41.970667 20091 catalog_manager.cc:1259] Loaded cluster ID: a0c9bb4b881d4170b25329c9cb60a629
I20260928 02:50:41.970758 20091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: loading cluster ID for follower catalog manager: success
I20260928 02:50:41.972796 20091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: acquiring CA information for follower catalog manager: success
I20260928 02:50:41.974087 20091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 199f8584466e43cd87b4a27a7ce37a60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:50:41.980911 20116 catalog_manager.cc:1259] Loaded cluster ID: a0c9bb4b881d4170b25329c9cb60a629
I20260928 02:50:41.981011 20116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: loading cluster ID for follower catalog manager: success
I20260928 02:50:41.982586 20116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: acquiring CA information for follower catalog manager: success
I20260928 02:50:41.983723 20116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c388365249e4cda99f4f50cefe0ab4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:50:42.219748 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:42.215927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.28.248.189:44411)
0928 02:50:42.216199 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:42.216203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:42.216218 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:50:42.216273 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:42.216275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:42.216418 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:42.216527 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:42.216795 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.217384 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.218223 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.218473 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.218570 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:42.218874 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:42.219319 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:42.219428 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:42.219477 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:50:42.219530 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260928 02:50:42.227062 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:42.223191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.28.248.188:34877)
0928 02:50:42.223473 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:42.223476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:42.223489 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:50:42.223599 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:42.223601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:42.223715 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:42.223811 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:42.224089 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.224742 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.225588 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.225814 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.225912 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:42.226159 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:42.226593 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:42.226685 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:42.226770 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:50:42.226819 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":121,"threads_started":1}
I20260928 02:50:42.233965 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:42.229873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.28.248.189:44411)
0928 02:50:42.230130 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:42.230134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:42.230452 (+ 318us) server_negotiation.cc:407] Connection header received
0928 02:50:42.230595 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:42.230598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:42.230675 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:42.230744 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:42.231077 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.231840 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.232789 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.233098 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.233199 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:42.233411 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:42.233502 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:42.233570 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:42.233699 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:50:42.233753 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260928 02:50:42.234133 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:42.230235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.28.248.188:34877)
0928 02:50:42.230343 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:42.230347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:42.230569 (+ 222us) server_negotiation.cc:407] Connection header received
0928 02:50:42.230751 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:42.230754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:42.230810 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:42.230857 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:42.231191 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.231683 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.232982 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.233297 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.233429 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:42.233691 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:42.233756 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:42.233819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:42.233955 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:50:42.234013 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:50:42.235359 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:50:42.229905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.28.248.190:33279)
0928 02:50:42.230276 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 02:50:42.230282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:50:42.230299 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:50:42.230395 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:50:42.230399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:50:42.230570 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:50:42.230692 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:50:42.230910 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.231766 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.233498 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:50:42.233860 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:50:42.234251 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:50:42.234410 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:50:42.234745 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:50:42.235128 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0928 02:50:42.235163 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:50:42.235219 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":68,"threads_started":1}
I20260928 02:50:42.238427 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19851
I20260928 02:50:42.245072 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 19926
I20260928 02:50:42.251011 29666 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9u0ndH/build/debug/bin/kudu with pid 20002
2026-09-28T02:50:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3125 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6232 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281703 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST